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PREFACE 


This  study  grew  out  of  a  concern  that  Department  of  Defense  (DoD)  and  Air  Force 
efforts  to  contain  their  risks  in  weapon  system*-  development  programs— for  exam¬ 
ple,  risks  that  costs  would  escalate,  performance  would  be  deficient,  or  that  a  pro¬ 
gram  would  be  canceled  because  its  development  was  too  expensive — were  resulting 
in  policies  shifting  inappropriate  risk  to  their  contractors  or  creating  other  unantici¬ 
pated  side  effects.  At  the  suggestion  of  Maj.  Gen.  lohn  Slinkard,  SAF/AQC,  this  study 
was  initiated  to  examine  the  way  in  which  risk  was  identified  and  managed  in  a 
group  of  recent  program  development  efforts.  The  purpose  was  to  develop  a  better 
empirical  basis  for  discussion  and  to  provide  senior  Air  Force,  DoD,  and  congres¬ 
sional  officials  with  an  overview  of  the  important  policy  issues. 

This  report  summarizes  and  integrates  the  findings  from  the  study,  while  the  major 
part  of  the  research  is  documented  in  other  RAND  publications: 

•  Susan  J.  Bodilly,  Case  Study  of  Risk  Management  in  the  USAF  B-IB  Bomber 
Program,  N-3616-AF,  1993. 

•  Susan  J.  Bodilly,  Case  Study  of  Risk  Management  in  the  USAF  LANTIHW  Program, 
N-3617-AF,  1993. 

•  Frank  Camm,  The  Development  of  the  FlOO-PW-220  and  FI  lO-GE-lOO  Engines:  A 
Case  Study  in  Risk  Assessment  and  Risk  Management,  N-3618-AF,  1993. 

•  Frank  Camm,  The  F-I6  Multinational  Staged  Improvement  Program:  A  Case 
Study  of  Risk  Assessment  and  Risk  Management,  N-36I9-AF.  1993. 

•  Kenneth  R.  Mayer,  The  Development  of  the  Aavanced  Medium-Range  Air-to-Air 
Missile:  A  Case  Study  of  Risk  and  Reward  in  Weapon  System  Acquisition,  ,\'-3620- 
AF,  1993. 

Two  related  case  studies  by  Timothy  J.  Webb  on  risk  management  during  the 
development  of  the  Global  Positioning  System  Block  I  Satellite  and  risk  management 
in  preparing  for  development  of  the  Joint  Surveillance  Target  Attack  Radar  System 
(Joint  ST.ARS)  are  as  yet  unpublished. 

The  study  was  initiated  before  the  Cold  War  seemed  so  completely  concluded  and 
deals  with  developments  that  largely  took  place  during  'he  1980s.  In  the  present 
policy  environment,  it  seems  unlikely  that  the  same  pressures  that  drove  system  ac¬ 
quisition  in  that  decade  will  remain,  However,  the  study  investigates  problems  that 
accompany  any  effort  by  large  organizations  to  manage  complex  and  risky  develop- 
rn ent  efforts  in  the  fiiw-  o*  ww.«rcc  c  *1ic  leciHiiq-jcs  tlvat 
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program  offices  used  to  manage  weapons  developments  seem  particularly  well- 
suited  to  the  weapons  acquisition  environment  of  the  1990s. 

This  study  leflects  an  understanding  of  acquisition  processes  derived  from  35  years 
of  research  on  research  and  development  and  acquisition  management  by  RAND, 
most  of  it  under  the  auspices  of  Project  AIR  FORCE.  The  lessons  derived  from  this  re¬ 
search,  which  were  summarized  some  years  ago  in  a  RAND  report  by  Michael  D. 
Rich,  Edmund  Dews,  and  C.  L.  Batten,  Improving  the  Militaty  Acquisition  Process: 
Lessons  from  HAND  Research  (R-3373-AF/RC,  February  1986),  are  generally  con¬ 
firmed  by  the  research  presented  here.  However,  the  emphasis  in  this  report  is 
somewhat  different.  The  earlier  research  dealt  with  acquisition  policies  that,  if 
implemented,  R/\ND  judged  would  lead  to  superior  outcomes.  The  research  here 
focuses  on  the  implementation  of  existing  policies.  The  authors  seek  to  show  how 
oiganizational  incentives,  political  pr'icesses.  and  communication  difficulties  shape 
the  implementation  of  these  policies,  oficn  in  ways  that  defeat  their  original  intent. 

This  rescarcli  was  sponsored  by  the  Deputy  Assistant  Secretary  for  Acquisition 
(Contracting)  of  the  United  States  Air  Force  as  part  of  the  Resource  Management  and 
Systems  Acquisition  Program  of  Project  ;\IU  FORCE.  This  report  is  intc'tuied  to  be  of 
interest  to  senior  officials  of  the  F)nl)  and  th.c  Congress. 
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SUMMARY 


Managing  the  development  of  a  major  weapon  system  is,  to  a  great  extent,  the  man¬ 
agement  of  risk.  Top  managers  in  the  Air  Force  and  its  contractors  will  aim  at  man¬ 
aging  development  in  such  a  way  that  the  risks  of  shortfalls  in  performance  or  unac¬ 
ceptable  development  times  and  costs  are  monitored  and  controlled. 

This  report  integrates  the  findings  of  seven  case  studies  of  major  weapon  system  de¬ 
velopment  efforts.  It  attempts  to  understand  how  the  actions  of  major  actors, 
including  the  System  Program  Office  (SPO).  supervising  command.  Air  i-orce 
Headquarters,  Department  of  Defense  (DoD),  the  contractors,  and  Congress,  interact 
to  shape  die  character  and  levels  of  risk  inherent  in  a  development  and  to  manage 
that  risk  throughout  the  development.  The  systems  that  are  subjects  of  the  case 
studies  were  chosen  to  represent  a  wide  spectrum  of  development  tasks  and  levels  of 
risk  and  include  the  following: 

1.  B-IB  Bomber 

2.  Advanced  Medium-Range  Air-to-Air  Missile  (AMRAAM) 

3.  Low-Altitude  Navigation  Targeting  Infrared  for  Night  System  (LrtNTlRN) 

4.  FlOO-PVV-220  and  FllO-GE-100,  Alternate  Fighter  Ensioes  (AFE) 

5.  Space  Elements  of  the  Global  Positioning  System  {(  ■  ■ 

6.  F-IG/Multinational  Staged  Improvement  Program  (MSiP) 

7.  Joint  Surv'eillance  Target  Attack  Radar  System  (Joint  STARS) 

The  studies,  written  largely  from  the  perspective  of  the  SFO,  examine  the  manage¬ 
ment  of  risk  during  full-scale  developm.ent  but  often  reach  back  to  the  imponant 
planning  and  development  activities  in  earlier  stages  of  the  acquisition  process. 

In  our  analysis,  we  grouped  the  seven  programs  into  three  categories.  Three,  the 
B-IB,  AMRAAM.  and  lAiNTlRN,  experienced  substantial  acquisition  difficulties 
during  full-scale  development.  Three  programs,  the  AFE,  GPS,  and  F-16  MSIP, 
proceeded  fairly  smoothly.  The  final  program.  Joint  STARS,  experienced  modest 
difficulties  and  turned  out  to  be  the  furthest  from  completion  at  the  time  of  our 
research.  This  categorization  was  based  only  on  the  management  difficulties,  not  on 
the  ultimate  outcome  or  value  of  the  program. 

The  three  programs  categorized  as  experiencing  significant  acquisition  difficulties 
share  a  number  of  important  characteristics. 
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1 .  Each  was  viewed  as  conlral  to  key  Air  Force  missions. 

2.  Al!  pos.scsscd  demanding  cost,  schedule,  and  performance  nbjociives  that  resulted 
from  two  important  pressures,  (1)  strong  demands  from  operational  commands, 
and  [2)  the  need  to  sell  the  program  to  authorities  outside  the  Air  Force. 

3.  All  included  a  high  degree  of  concurrency  hetsvecn  development  and  production. 

4.  Each  was  subjected  to  important  constraints  (funding  caps,  mandated  .schedules, 
etc.)  that  limited  the  capacity  of  the  program  office  to  manage  the  program. 

.5.  In  each,  there  was  lack  of  experti.se,  personnel,  and  continiiity  in  the  SPO  at  some 
key  points  in  die  development  process. 

6.  For  a  variety  of  reasons,  top-level  management  oversight  in  the  Air  Force  and  DoD 
discounted  lower-level  assessments  of  the  level  of  risk. 

7.  For  cacli,  there  were  important  "holes"  in  the  technology  base,  which  were  rec¬ 
ognized  but  de-emphasized  in  the  advocacy  for  the  program. 

0.  In  two  of  tlie  three  programs,  ilie  contractor  bought  into  llie  program  expecting  in 
recoup  eariy  iosses  tlirough  tire  prosjira  i  of  long-term  sales  that  became  increas- 
inglv  problem;', tic. 

In  contrast,  the  programs  that  proceeded  fairly  smoothly  from  an  acquisition  man¬ 
agement  viewpoint  were  characterized  by: 

1.  Technologies  tliat  were  well  in  hand  or  handled  effectively  in  earlier  phases  of  the 
development  process. 

2.  Modest  levels  of  concurrency. 

3.  Flexibility  to  make  schedule,  cost,  and  performance  tradeoffs. 

4.  Strong  and  technically  qualilled  program  offices  that  possessed  significant  ele¬ 
ments  of  staff  continuity'. 

5.  Relatively  little  external  oversigirt. 

Perliaps  just  as  significant,  tlierc  were  a  number  of  features  of  tiiese  progranns  tliat  do 

not  r/;7)er  systematically  across  these  programs,  featuies  ili<i!  are  important  subjects 

of  acquisition  management  policy  for  the  devclopmcm  piiasc.  In  particular: 

1.  There  were  no  systematic  differences  in  the  types  of  cOiitracis  used.  Al!  used 
nxed-price  contracts  during  development;  several  used  firm  nxed-pricc  contracts. 
But  the  terms  of  these  contracts  varied  wideiy  hi  areas,  such  as  the  share  line, 
ceiling,  and  warranty. 

2  Most  had  a  consistent  requirement  tluough  development,  altliough  Joint  S’I'ARS 
had  requirement  instability  due  to  dilfiruhies  in  resolving  .Air  Force  and  ,'\rniy 
differences  roncerning  the  operational  concept  for  using  the  system. 

3.  There  was  a  spectrum  of  source  selection  procedures  ranging  from  sole  so'urco  to 
sustained  competition,  but  tlie  procedures  did  not  seem  consi.stcntly  related  to 
acquisition  management  outeonics. 
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4.  The  developments  did  not  differ  in  the  timing  of  when  technological  problems 
that  faced  developers  in  full-scale  development  (FSD)  became  known.  In  all  cases 
these  problems  were  identified  either  very  early  in  FSD  or  before. 

5.  All  (except  the  B-lB]  dealt  with  significant  externally  imposed  budget  changes  in 
the  course  of  tlreir  developr  ients. 

Why  did  our  systems  differ  in  their  apparent  ability  to  manage  risk  effectively  and  to 
move  through  the  acquisition  process?  On  the  basis  of  our  field  work,  we  conclude 
that  the  key  ingredients  of  successful  risk  management  are  qualified  technical  staff, 
possessing  sufficient  flexibility-  (or  slack)  to  respond  to  both  expected  or  unantici¬ 
pated  difficulties  in  the  program,  coupled  with  oversight  that  is  qualified  and  that 
forces  consideration  of  progrrm  issues  that  are  beyond  the  purview  of  the  program 
managers.  The  programs  that  experienced  difficulties  did  not  consistently  possess 
these  ingredients.  Those  programs  experiencing  difficulty  were  relatively  large  and 
important  to  key  Air  force  interests.  Advocacy  required  to  advance  them  through 
the  system  led  to  early  and  often  overly  optimistic  promises  concerning  their 
outcomes.  They  were  so  important  ,  the  Air  Force  that  it  wa.s  willing  in  some 
instances  to  agree  to  fui.tling  levels,  schedule  gu  -antces,  .  nd  ‘‘unding  caps  that  were 
unrealistic.  These  squeezed  out  the  program  slack  needed  ‘‘or  effective  risk  man¬ 
agement. 

In  contrast,  the  more  managertally  successful  systems  we  examined  did  not  require 
the  same  level  of  promises.  They  tended  to  be  far  less  visible  up  the  management 
line.  The  MSIP  and  the  ,AFE  both  evolved  out  of  strong  and  highly  organized  SPOs. 
The  GPS  was  planned  by  a  lec.inically  qualified  program  office  that  enjoyed  a  special, 
personal,  and  direct  relationship  with  the  office  of  the  Director,  Defense  Research 
and  Engineering  (DDR&E):  this  seems  to  have  been  possible  because  the  program 
was  not  absolutely  front  and  center  on  the  Air  Force’s  agenda.  The  programs 
proceeded  with  comparatively  broad  specifications,  so  that  their  managers,  when 
confronted  with  problems,  could  make  tradeoffs  they  felt  benefited  the  entire 
system. 

It  is  important  to  emphasize  we  found  nothing  to  suggest  that  the  basic  policies 
governing  the  management  of  large-scale  developments  are  fundamentally  flawed. 
Rathe '  ,  rhe  problems  came  in  fiieir  implementation.  Realistic  cost,  schedule,  and 
performance  goals,  coupled  with  quality  personnel,  are  the  keys  to  good  risk  manage¬ 
ment.  WTierc  significant  acquisition  problems  arose  in  the  cases  we  .U'.'iied,  they 
were  usually  associated  with  schedules  that  were  overly  ambitious,  budgets  that 
provided  no  funds  for  contingencies  and/or  hedging,  or  a  reluctance  to  adjust  per¬ 
formance  goals  after  the  consequences  of  maintaining  those  goals  became  apparent. 
The  reasons  for  this  appear  rooted  in  the  decisionmaking  processes  of  large  bureau¬ 
cracies,  decisionmaking  processes  that  provide  strong  incentives  for  optimistic 
promises  and  disincentives  for  full  disclosure  of  problems.  The  most  important  mes  ■ 
sage  of  this  study  is  that  the  senior  officials  in  the  Air  Force  and  the  DoD  who  are 
charged  with  making  decisions  concerning  strategies  tor  developing  new  systems  as 
well  as  setting  guidance  for  schedule,  cost,  and  performance  targets  should  temper 
recommendations  brought  forth  by  their  subordinates  with  an  understanding  of  the 
intense  advocacy  environment  that  frequently  shapes  those  recommendations. 
Moreover,  they  should  also  recognize  that  sometimes  their  own  behavior  shapes  that 
of  their  subordinates. 


xll  Barrierj  to  Managing  Risk 


To  this  brond  conclusion,  we  would  add  two  more  limited  suggestions.  First,  two  of 
the  programs  we  examined,  the  AFE  and  the  F-16  MSIP,  seem  to  us  to  provide  useful 
guidelines  for  research  and  development  management  for  a  significant  proportion  of 
the  Air  Force’s  acquisition  program.  The  long-term  continuity  of  the  program  office, 
use  of  matrixed  program  and  technical  staff,  sustained  and  flexible  contracting,  and 
evoiiition  of  technologj’-specific  development  doctrine  appeared  to  contribute  to 
successful  development  outcomes. 

Second,  several  of  the  programs  in  our  study  could  be  said  to  have  gotten  "out  of 
control"  in  that  they  were  subjected  to  a  continuing  series  of  reviews,  site  visits,  pro¬ 
gram  revisions,  congressional  interventions,  and  personnel  changes  that,  in  our 
judgment,  often  seemed  to  have  deleterious  effects  on  the  program  outcome.  VVe 
think  that  many  of  the  actions  taken  by  complex  and  political  bureaucracies  when  a 
program  is  in  deep  trouble  are  counterproductive  because  th  /  constrain  program 
managers  in  making  needed  changes.  The  Air  Force  and  DoD  should  consider  some 
form  of  special  management  for  programs  that  are  experiencing  great  difficulty — 
something  akin  to  putting  a  business  into  receivership 
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A/C 

Aircraft 
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Alternate  fighter  engines 

AFSC 

Air  Force  Systems  Command 
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Airborne  instrument  laborator>’ 

AMRMM 

Advanced  Medium-Range  Air-to-Air  Missile 

APRFiP 

AMRAAM  Produdbilily  Enhancement  Program 

ARPA 

Advanc  ed  Research  Projects  Agency 

ATI' 

Advanced  tactical  tighter 

ATR 

Automatic  target  recognizer 

DDH&H 

Director,  Defettse  Researcii  Lngineci  ing  (Ollicc  of  ibiC) 

dem/val 

Demonstration/ validation 

DoD 

Department  of  Defense 

DSARC 

Defense  System  Acquisition  Review  Council 

FFP 

Firm  fixed-price 

FPI 

Fixed-price  incentive 

FSD 

Full-scale  development  (now  called  Engineering  and  Manufacturing 
Development,  EMD) 

FTI 

Fixed-target  indicator 

GD 

General  Dynamics 

GE 

General  Electric 

GPS 

Global  Positioning  System 

IOC 

Initial  operational  capability  'date  of 

IR&D 

Independent  research  and  development 

jPC 

loint  Program/Projert  Office 

loin  (STARS 

loint  Surveillance  Target  Attack  Radar  System 

bXNTIRN 

Low-Altitude  Navigation  Targeting  Infrared  for  Night  System 

LRIP 

Low-rate  initial  production 

MLRS 

Multiple-launch  rocket  system 

MOA 

Memorandum  of  Agreement 

MOU 

Memorandum  of  Understanding 

MSIP 

Mulfina.tional  Staged  Improvement  Program 

MTI 

Moving-target  indicator 

0MB 

Office  of  Management  and  Budget 

OSD 

Office  of  the  Secrctar\'  of  Defense 

PM 

Program  manager 

P&W 

Pratt  and  Whitney 

R&D 

Research  and  development 

SPO 

System  Program  Office 

USAF 

U  nited  States  Air  Force 

_ Chapter  Oiv: 

INTRODUCTION 


The  development  stages  uf  a  major  weapon  system  acquisition  are  inherently  con¬ 
cerned  with  the  management  of  risk.  To  be  sure,  much  effort  must  be  devoted  to  un¬ 
derstanding  the  nature  of  the  operational  needs  that  give  rise  to  the  development,  in¬ 
venting  the  system  concepts  that  promise  to  meet  those  needs,  organizing  personnel 
to  design,  test,  and  fabricate  hardware  and  software  ,iiai  implement  the  concept, 
creating  the  capability  to  produce  the  system,  and  designing  the  training  and  main¬ 
tenance  support  needed  to  field  the  weapon  system.  But  the  top  managers  in  the  .'\ir 
F'orcc  and  its  contractors  will  be  concerned  with  managing  the  program  in  such  a 
way  that  the  risks  Tf  shontalls  in  performance  or  unacccptahle  development  times 
and  costs  are  monitored  and  controlled. 

This  report  integrates  the  findings  of  seven  case  studies  of  major  weapon  system  de¬ 
velopment  efforts  (see  Table  1).  It  seeks  to  understand  how  the  actions  of  major 
actors,  including  the  system  program  office  (SPO),  supervising  command.  Air  Force 
Headquarters,  Department  of  Defense  (DoD),  the  contractors,  and  Congress  interact 
to  shape  the  character  and  levels  of  risk  inherent  in  a  development  and  manage  that 
risk  throughout  the  development.  The  systems  that  are  subjects  of  the  case  studies 
were  chosen  to  represent  a  wide  spectrum  of  development  tasks  and  levels  of  risk. 
The  studies,  wTitten  largely  from  the  perspective  of  the  program  office,  examine  the 
management  of  risk  during  full-scale  development  but  often  reach  back  to  the 
important  planning  and  development  activities  in  earlier  stages  of  the  acquisition 
process. 

The  stor\'  we  tell  is  a  complex  but  common  one.  The  manner  in  which  levels  of  risk 
are  set  and  managed  is  ^^etermined  to  only  a  limited  degree  by  such  policies  as  those 
embedded  in  the  DoD  50on  scries  of  directives  and  regulations.  To  a  much  greater 


Tabic  I 

Programs  Selected  for  Study 
frogram 

1.  I)-lHHoniber 

2.  Advanced  Medium-ttange  Air-:o-Air  Missile  (AMICAAM: 

1  I.mv-.Altitude  Navigation  fargeiing  Infrared  for  Night  System  d.AN'l  iUN) 
‘1.  riOO  PW-22n  and  Pi  1C  GI--10U,  Alternate  t-ighier  P.ngires  lAPPi) 

5.  Space  Hiement  of  the  Ciioba'i  Positioning  Sy.stem  l(ii’S) 

6.  '.6.''Mtiln;iulii)nai  Stageef  improvement  Program  (MSIP; 

7.  Joint  Sorseiiiance  Target  Attack.  Radar  System,  (kiint  STAIiSl 
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extent,  the  level  and  management  ot  risk  is  a  result  of  complex  Interactions  among 
many  actors  and  organizations,  their  operating  procedures,  and  the  nature  of  the 
technology  Itself,  The  problems  that  the  Air  Force  and  the  DoD  have  with  managing 
risks  are  often  systenv':  problems  not  policy  problems.  The  normative  \new  of  risk 
management  embodied  in  DoD  Directive  5000,1  seems  solid  and  well-founded. 
However,  as  we  are  finding  in  many  other  areas  of  our  society,  improved  perfor¬ 
mance  of  important  private  and  governmental  functions  requires  an  understanding 
of  the  full  organizational  and  political  systems  as  well  as  a  uillingness  to  deal  simul¬ 
taneously  with  the  many  determinants  of  the  performance  of  that  system.  We  can¬ 
not  provide  such  a  full  understanding  and  array  of  proposed  actions  from  the  re¬ 
search  effort  to  be  described  here.  Rather,  the  goal  of  this  study  is  more  limited:  to 
contribute  to  our  understanding  of  the  system  itself  and  to  the  appreciation  that 
broad  prescriptive  policies  so  often  preferred  by  policymakers  may  have  little  effect 
in  the  face  of  the  myriad  incentives  built  into  our  complex  government  and  business 
organizations.' 

THE  VIEW  OF  RISK  USED  IN  THIS  STUDY 

For  the  purposes  of  this  study,  we  have  a  vcr\-  straightforward  concept  of  risk.  From 
the  point  of  view  of  top-level  management,  risk  is  the  probability  that  program  out¬ 
comes  occur  that  have  seriously  adverse  consequences  for  the  nation.  In  terms  of 
the  traditional  metrics  of  acquisition  management  (costs,  schedule,  and  perfor¬ 
mance),  risk  is  the  probability  that  costs  will  be  so  high  that  few  of  the  systems  can  be 
acquired  or  that  the  schedule  is  delayed  so  that  capabilities  needed  by  military  forces 
to  meet  perceived  threats  will  not  exist  in  a  timely  fashion,  or  that  the  performance  of 
the  system  will  not  allow  the  forces  to  meet  that  threat. 

WTiile  in  real  life,  measures  of  cost,  schedule,  and  performance  are  complex  and 
multidimensional,  intuitively  we  can  represent  this  notion  of  risk  for  a  single  out¬ 
come  dimension  quite  simply  as  we  do  in  Figure  1.  This  is  a  picture  of  a  subjective 
probability  distribution  of  some  important  outcome  held  by  leaders  of  a  program.  In 
this  diagram,  the  outcome  value  0  represents  a  level  below  W'hich  the  performance  is 
deemed  seriously  inadequate  for  the  mission,  and  the  shaded  area  is  the  probability 
that  such  inadequate  performance  will  be  obtained. 

The  level  of  risk  can  be  changed  in  a  number  of  ways.  For  example,  additional  re¬ 
sources  might  be  added  to  allow  the  project  to  sustain  two  approaches  to  achie\ang 
the  performance.  The  joint  probability  that  both  approaciies  fail  to  meet  the  perfor¬ 
mance  level  0  should  be  less  than  that  of  a  single  approach;  the  distribution  would 
be  moved  to  the  right  (and  changed  in  shape)  so  the  shaded  area  decreases. 
Alternatively,  the  designers  and  operators  might  take  actions  in  other  parts  of  the 
system  or  revise  their  operational  concept  so  that  the  outcome  O  is  less  critical  and 
the  mission  can  be  achieved  at  a  lesser  level  of  performance.  The  vertical  line  at  0 
moves  to  the  left,  and  the  risk  is  reduced. 


'ihf  Tieldwork  for  the  case  studies  was  done  in  late  1990  and  early  1991 .  The  cases  themselves  deal  with 
Air  Force  and  contractor  operations  as  they  existed  roughly  between  1975  and  1990.  Since  tins  time,  a 
profound  restructuring  of  the  ,Air  Force  has  been  initiated.  Surely,  these  new  organizations  would  manage 
programs  such  as  we  describe  differently  (and  perhaps  better;.  We  remain  persuaded,  however,  that  the 
general  phenometion  we  describe  and  the  perspective  we  take  remain  relevant  in  this  new  era. 
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Figure  1 — Representation  of  Risk 


This  simple  representation  requires  qualification  in  two  ways  for  our  needs.  First, 
the  number  of  outcomes  that  arc  properly  of  concern  is  enormous  and  estimations  of 
the  type  of  distribution  shown  in  Figure  1  are  not  normally  possible.  For  very  simple 
cases,  formal  analytic  techniques  exist  to  create  such  distributions,  Such  techniques 
may  be  used  to  provide  inputs  to  program  planning,  but  managers  seldom  rely  ex¬ 
clusively  on  them.  Rather,  when  dealing  with  programs  as  a  whole,  managers  prefer 
to  adopt  robust  strategies  that  drive  the  risks  of  catastrophic  downside  events  to  an 
acceptably  low  level.  This  involves  a  good  deal  of  organizational  attention  to  risk  as¬ 
sessment  but  not  generally  a  reliance  on  formal  analytic  techniques, 

Second,  this  discussion  has  assumed  that  the  risk  is  assessed  by  a  rational  actor  who 
has  a  clear  understanding  of  the  national  interest.  In  fact,  many  actors,  including 
members  of  the  program  office,  devt’.  rpment  and  using  commands.  Air  Force  head¬ 
quarters,  DoD,  Congress,  and  the  contractor  community',  are  involved  in  planning 
and  other  acmities  that  set  the  risk  surrounding  a  weapon  system  development  and 
affect  the  capacity  to  manage  tnat  risk.  These  actors  vary  in  the  dimensions  of 
dou'nside  risk  tliat  they  emphasize  and  in  their  knowledge  of  the  factors  that  deter¬ 
mine  that  risk.  The  technical  staff  of  a  contractor  may  have  a  traditional  concern  for 
the  failure  of  the  system  to  achieve  performance  levels  within  available  budget.  For 
the  contractor's  top  management,  the  risks  may  be  failing  to  win  a  competition  or 
the  possibility  of  a  cancellation  of  an  e.xisting  program  that  will  have  effects  on  the 
corporation's  sales  and  profits.  Air  Force  leaders  may  emphasize  the  risk  of  poor  or 
untimely  performance  or  of  termination  of  a  program.  The  Congress  may  be  con¬ 
cerned  over  program  outcomes  that  injure  their  constituents  or  affect  budgetary 
decisions  they  have  previously  made.  The  program  office,  the  focal  point  of  die 
government's  risk  assessment  and  management  activities,  shares  many  of  the  per¬ 
ceptions  of  these  otlier  actors,  but  its  indhidual  staff  members  (along  with  many  of 
the  contractor's  stafO  may  also  factor  in  concerns  for  adverse  risks  related  to  career 
advancement. 

The  concerns  of  these  actors  are  legitimate  and,  our  cases  show,  often  affect  the  di¬ 
mensions  of  risk  that  receive  emphasis,  the  levels  of  that  risk,  and  the  capability'  of 
the  contractors  and  program  office  to  manage  that  risk.  In  a  costly  program  central 
to  Air  Force  core  missions,  for  example,  negotiations  will  take  place  between  the  Air 
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Force  and  the  DoD  and  the  Congress  concerning  cost,  performance,  and  schedule 
levels.  To  maximize  the  probabilit>’  of  program  approval,  the  Air  Force  has  incentives 
to  represent  Its  program  in  tire  best  possible  light.  It  will  want  to  promise  high  levels 
of  performance  and  timeliness,  coupled  with  reasonable  costs,  to  persuade  its  supe¬ 
riors  that  the  system  should  be  approved  or  continued.  The  Congress  or  DoD,  on  the 
other  hand,  recognizing  the  incentives  driving  the  Air  Force,  arc  concerned  that  the 
promises  may  be  optimistic  and  that  the  risks  of  failure  to  meet  the  promises  are 
high.  The  actual  choices  of  key  outcomes  are  the  result  of  this  complex  bargaining 
process  involving  actors  with  a  variety  of  perspectives  and  objectives.  Our  analysis 
suggests  that  under  some  circumstances,  this  bargaining  process  has  the  effect  of  in¬ 
creasing  the  levels  of  risk  to  the  nation  of  ineffective  acquisitions.  It  does  this  both  by 
increasing  the  outcome  thresholds  that  are  used  to  judge  whether  the  acquisition  is 
seriously  deficient  and  by  reducing  the  time  and  resources  available  to  program 
managers  to  deal  with  threats  to  outcome  shortfalls. 

This  is  easily  illustrated  by  Figure  1.  Suppose  that  the  outcome  dimension  is  some 
measure  of  payload/range/penetration  capability  for  a  new  bomber.  Given  some  es¬ 
timate  of  the  threat,  advocates  of  the  system  have  cver\’  reason  to  promise  the  best 
performance  measure  possible— to  ir\'  to  move  the  value  Oto  the  right,  hut  at  the 
same  time,  there  will  be  pressures  to  show  that  the  costs  for  achieving  this  capability 
are  reasonable  and  that  the  capability'  can  be  obtained  in  a  timely  way.  Both  these 
pressures  w-ill  tend  to  shift  the  distribution  of  outcomes  in  Figure  1  to  the  left,  be¬ 
cause  fewer  resources  will  be  available  to  cope  with  technical  difficulties,  and  quite 
possibly,  greater  levels  of  concurrency  between  development  tasks  and  development 
and  production  vsill  reduce  the  time  that  the  developers  have  to  deal  with  unex¬ 
pected  development  outcomes. 

These  observations  are  scarcely  new.  The  incentives  we  describe  are  w’ell-known, 
and  Air  Force  and  DoD  procedures  intended  to  modify'  and  cope  witlt  them  have 
long  been  in  place.  Most  prominent  are  the  DoD  5000  series  directives  and  regula¬ 
tions.  These  directives  specify  an  acquisition  process  whose  broad  features  have  re¬ 
mained  unchanged  for  niany  years.  The  principal  phases  of  the  process  are  shown  in 
Figure  2. 

A  formal  concept  formulation  phase  focusc'  on  the  mi.ssion  needs  and  the  alterna¬ 
tive  operational  and  technological  approaches  to  meeting  those  needs.  The  demon¬ 
stration /validation  (dem/val)  phase  is  used  to  try  out  critical  components  of  the  sys¬ 
tem  and  often  involves  tests  of  prototypes  of  key  system  components.  Only  when  the 
risks  have  been  reduced  by  both  these  development  and  test  activities  and  continu¬ 
ing  mission  analysis  is  the  program  supposed  to  be  approved  for  entry  into  full-scale 
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development  (FSD).^  These  5vo  phases  are  expected  to  have  costs  that  arc  modest 
compared  with  those  of  full-scale  development  and  to  reduce  the  risks  of  FSD  sub¬ 
stantially  by  sharpening  the  understanding  of  desirable  outcome  targets  and 
identifying  areas  of  technical  risk  that  must  be  specifically  addressed  during  this 
critical  phase  of  actKdty. 

The  goal  of  our  study,  then,  is  to  understand  better  how  risk  is  identified  and  man¬ 
aged  during  these  two  critical  development  phases  and  to  suggest  ways  in  which  that 
process  might  be  improved. 

ORGANIZATION  OF  REPORT 

The  empirical  basis  for  our  findings  is  in  the  case  studies.  Chapter  Two  outlines  fac¬ 
tors  that  we  anticipated  would  affect  the  level  of  risk  and  the  capability  of  the  pro¬ 
gram  office  and  contractors  to  manage  that  risk.  It  describes  how  wc  chose  the 
systems  for  study  and  briefly  describes  those  systems.  (The  studies  themselves  arc 
published  separately.)  Chapter  Three  summarizes  what  we  found  concerning  how 
factors  affecting  ri.sk  were  treated  in  our  cases  and  providc.s  an  overall  analysis  that 
serves  as  tire  ba.sis  for  ih''  findings  and  recommendations  contained  in  Cliapior  Ifiuir. 


^Full-Scale  Development  Is  now  called  Engineering  and  Manufacturing  Development  (HMD),  but  the 
earlier  terminology  is  retained  in  this  report. 


_ _ _ _ _ Chapter  Two 

MANAGING  RISK  IN  SEVEN  SYSTCIvToEVELOPMENTS 


STUDY  DESIGN’  AND  SELECTION  OF  CASES 

Our  study  design  uses  seven  case  studies  of  weapons  acquisition  to  examine  the 
manner  in  which  risk  was  dealt  witli  during  their  development.  On  the  basis  of  pre¬ 
vious  rcscarcli  and  our  review  of  the  literature,  we  expected  ili.n  a  number  of  factors 
would  affect  botli  the  level  of  risk  at  various  stages  of  the  program  iintl  tlie  ctipticity  of 
the  .Air  I'orec  and  the  contractor  to  manage  that  risk.  In  paiiicuhir,  we  iileniilied  the 
following; 

1.  The  degree  to  which  the  technologx'  underlying  the  development  is  well  in  hand  at 
the  initiation  of  full-scale  development. 

2.  The  stability  of  the  requirement  for  the  system  through  the  development. 

3.  The  qualitlcadons  and  continuity  of  experience  of  key  members  of  the  SPO  staff. 

4.  Features  of  the  program  management  strategy,  such  as  the  level  of  concurrency. 

5.  The  nature  of  the  contractual  relations  between  the  contractor  and  the  govern¬ 
ment. 

6.  The  prominence  and  nature  of  intra-govornmcntal  or  imcr-scrvice  involvement  in 
the  program. 

7.  The  use  of  competition  throughout  the  program. 

8.  The  nature  of  the  contractors. 

9.  The  nature  of  constraints  external  to  the  program  created  by  Congress,  the  DoD, 
or  I-leadquarters  United  States  Air  Force  (USAF). 

The  case  studies  examined  these  factors  and  aiicmptcd  to  understand  how  they  in¬ 
teracted  to  affect  program  outcomes. 

In  selecting  cases  for  study,  we  wanted  a  spccirum  of  development  situations.  We 
sought  to  look  at  a  variety  of  technologies  and  types  of  systems  rather  than  restricting 
ourselves  to  aircraft,  missiles,  or  space  systems.  We  wanted  to  examine  both  large 
and  small  programs;  politically  controversial  programs  as  well  as  ones  about  which 
considerable  consensus  existed.  We  thought  it  important  to  look  at  programs  with 
low  expected  production  volumes  ns  well  as  those  for  which  long  production  runs 
were  anticipated.  Finally,  we  wanted  to  look  at  relatively  recent  programs,  both  be¬ 
cause  they  would  have  been  conducted  under  acquisition  policies  iliat  were  similar 
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to  tliose  existing  as  we  initiated  our  study  and  because  we  felt  data  and  people  would 
be  more  readily  available.  Thus  we  tried  to  examine  programs  that  were  initiated  in 
the  r.ud-l970s  or  later.  Table  2  lists  the  systems  that  we  examined. 

The  history  of  each  of  these  programs  is  complex  and  idiosyncratic,  as  is  alvvays  the 
case  wdth  research  and  development  activities.  The  conduct  and  outcomes  depend 
in  no  small  measure  on  personalities,  luck,  ar.d  the  nature  of  the  political  environ¬ 
ment  in  which  die  program  is  carried  out.  Our  case  studies  try  to  capture  some  of 
this  richness  and  must  be  read  to  understa-  ^  the  specifics  of  each  case.  However, 
the  task  of  this  report  is  to  try  to  step  back  from  these  individual  features  and  ask 
what  the  collection  of  cases  as  a  whole  tells  us  about  acquisition  management  and 
more  particularly  the  management  of  risk.  To  facilitate  this,  we  provide  only  brief 
overview  descriptions  of  the  programs  here. 


B-IB  BOMBER 

The  B-IB  case  concerns  the  procurement  of  an  entire  platform  and  its  component 
systems.  In  1981,  President  Reagan  announced  plans  to  buy  a  fleet  of  100  strategic 
bombers  to  replace  the  aging  U-52  force.  This  decision  was  the  culmination  of  a 
decades-long  debate  over  whether  (and  with  what)  to  replace  the  aging  bomber 
force,  At  the  same  time  that  the  B-IB  bomber  program  was  announced,  the  Air  Force 
also  undertook  the  development  of  a  stealth  bomber,  the  B-2,  to  replace  the  B-IB. 

The  B-IB  was  designed  wnth  many  improvements  over  the  B-52.  It  was  stealthier  be¬ 
cause  its  radar  cross  section  was  significantly  less  visible  to  enemy  radar  than  the  old 
B-52.  The  design  incorporated  improved  offensive  and  defensive  avionics  to  combat 
the  increased  Soviet  capabilities. 


Table  2 

Systems  Selected  for  Case  Studies 


Type  of  System 

Size  of  Program 
(Billions  of 
\  BAR  $) 

Ratio  of  R&D  to 
I’roc'  iction 
Ousts  i%) 

B-IB  Bomber 

Advanced  Medium-Range  .Air-to-Air 

Aircraft 

811,7 

14 

Missile  (AMRAAM) 

Low-Adtitude  Navigation  Targeting 
Infrared  for  Night  System 

Missile 

6.(1 

21 

(UNTIRN) 

F100-PVV.220and  FUO-GK-lOO, 

•Avionics 

3. .8 

id5 

.Alternate  Fighter  Engines  (AFbj 
Global  Positioning  System 

•Aircraft  engine.s 

>10.0 

<7 

(Space  Elements)  (GPS) 
F-16/Multinational  Staged 

Space  system 
Modification 

2.7 

149 

Improvement  Program  (MSIP) 

Joint  Surveillance  Target  Attack 

program 
Command  and 

24.0 

4 

RadarSystem  (Joint  .STARS; 

control 

7.1 

Hi 

NOTE;  Figures  are  approximate  and 

include  estirrates,  because  production 

is  not  Finished  for 

most  systems. 
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The  decision  to  procure  the  B"1B  was  made  by  the  President,  in  consulta‘-'^n  with  the 
Depanment  of  Defense,  and  by  the  Congress  after  negotiations  over  the  quantity, 
cost,  and  schedules  for  procurement.  Rapid  development  was  essential.  At  the  time 
the  program  was  undertaken,  some  defense  planners  felt  the  United  States  would  be 
vulnerable  to  a  Soviet  counterforcs  attack  without  the  B-IB  bomber.  These  planners 
talked  of  a  window  of  \ulnerability  that,  without  the  B-IB,  would  leave  the  United 
States  open  to  a  devastating  attack  by  superior  Soviet  strategic  forces.  Congress  and 
the  President  were  convinced  of  the  need  for  a  bomber  to  guard  against  this  attack, 
but  Congress  was  concerned  about  the  cost  of  the  procurement. 

The  high-level  negotiation,  outside  of  the  controi  of  the  SPO,  culminated  in  a  pro¬ 
gram  management  plan  with  severe  constraints.  Both  branches  of  government 
agreed  to  a  firm  and  urgent  date  of  initial  operational  capability  (IOC)  requiring  ex¬ 
tensive  concurrency,  a  funding  cap,  a  multi-year  contract,  and  an  additional  role  for 
the  SPO  to  act  as  the  system  integrator  to  reduce  the  costs  of  the  procurement.  Thi.s 
decision  was  based  on  the  assumption  that  technical  risk  was  low  because  the  con¬ 
tractors  had  previously  developed  the  B-IA. 

This  assumption  proved  to  be  false.  The  B-l.A  had  not  been  fully  tested  and  devel¬ 
oped  when  its  program  was  canceled.  I-urlher  tests  showed  significant  problems  that 
would  have  to  be  addressed  in  the  B-IB  program.  Moreover,  the  B-IB  design  called 
for  improvements  based  on  untested  technology  in  several  areas,  the  most  important 
of  which  was  the  defensive  avionics.  When  technical  difficulties  in  this  and  other 
systems  began  to  appear,  the  program  office  was  so  constrained  by  the  negotiated 
program  goals  that  many  remedial  actions  were  precluded. 

The  program  had  elements  of  both  success  and  failure.  It  met  its  nominal  funding 
cap  and  schedule  but  failed  to  meet  the  original  performance  requirements,  In  par¬ 
ticular,  it  failed  to  meet  the  performance  levels  required  during  the  predicted  win¬ 
dow  of  vulnerability. 

ADVANCED  MEDIUM-RANGE  AIR-TO-AIR  MISSILE  (AMRAAM) 

The  AMRAA.M  program  was  initiated  in  response  tn  Air  Force  studies  of  air-to-air 
combat  that  indicated  a  need  for  a  significant  improvement  over  the  capabilities  of 
the  current  medium-range  missiles  in  the  Air  Force  inventory'.  The  major  innovation 
in  AMRAAM  was  a  fully  active  guidance  system  that  would  give  AMRA/XM  a  true 
"launch  and  leave"  capability.  This  w'ould  free  pilots  from  the  need  to  illuminate  the 
target  aircraft  with  their  own  radar  (as  they  had  to  do  with  the  AlM-7  Sparrow,  which 
incorporated  a  semi-active  guidance  system),  and  permit  multiple  engagements  in  a 
single  pass.  AMRAAM  would  also  be  superior  to  the  AIM-7  against  low-altitude 
targets  and  electronic  countermeasures.  The  Air  Force  projected  that  AV1RAAM 
would  achieve  twee  the  combat  capability  of  the  AIM-7  at  one-half  the  cost. 

The  program  encountered  serious  technical,  cost,  and  schedule  difficulties,  largely 
related  to  the  failure  of  anticipated  technologies  to  mature  as  planned.  In  addition, 
external  demands  placed  on  the  program  hamper  the  ability  of  managers  to  respond 
to  development  problems.  The  Air  Force  Systems  Command  (AFSC)  shortened  the 
development  schedule,  which,  in  turn,  dictated  an  extremely  high  level  of  concur¬ 
rency,  and  required  that  a  second  source  be  qualified  early  in  the  development 
phase.  The  Office  of  the  Secretary  of  Defense  (OSD)  mandated  tliat  rigid  test  goals 
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and  thresholds  be  established,  and  congressional  requirements  stipulated  that  the 
program  would  have  to  meet  all  of  the  original  performance  specincations.  In  re¬ 
sponse  to  cost  increases,  the  Air  Force  initiated  an  extensive  redesign  effort  in  1985. 
tlie  AMRAAM  Producibility  Enhancement  Program  (APREP),  designed  to  reduce  pro¬ 
gram  costs. 

Though  AMRAAM's  cost,  despite  APREP,  is  still  much  higher  than  the  initial  esti¬ 
mates  (by  a  factor  of  three  to  five),  and  IOC  was  achieved  five  years  later  than  origi¬ 
nally  anticipated,  the  missile  has  met  performance  expectations. 


LOW-ALTITUDE  NAVIGATION  TARGETING  INFRARED  FOR  NIGHT 
SYSTEM  (LANTIRN) 

The  LANTIRN  is  an  avionics  system  providing  single-seated  fighters  with  the  capa- 
bilit\'  to  navigate  and  target  at  night  and  during  poor  weather.  At  its  initiation,  it  was 
expected  that  Ij\NT1RN'  would  dramatically  increase  the  capabilities  of  the  tactical 
air  forces.  As  such,  it  wa.s  viewed  as  an  urgent  requirement  by  the  Air  Force. 

The  early  requirements  imposed  technically  demanding  spccificuiions  for  system 
performance.  Enginccr.s  involved  in  the  program  described  the  technical  advances 
needed  to  meet  these  standards  as  high  risk  and  high  cost.  However,  because  of  the 
urgency  of  the  requirement  and  the  assertion  that  technical  problems  had  been  dealt 
with  in  an  earlier  “special-access"  program,  LANTIRN  proceeded  without  a  dem/val 
stage  to  test  its  technical  concepts  prior  to  the  commitment  to  procurement.  Our  re¬ 
search  suggests  that  advocates  for  the  program  inadequately  acknowledged  these 
risks  when  discussing  the  program  witli  Congress.  They  described  the  program  as  a 
low-risk,  low-cost  program,  setting  up  unrealistic  expectations  in  congressional 
sponsors, 

The  failure  to  recognize  fully  the  risk  involved  translated  into  an  ambitious  manage¬ 
ment  plan  that  called  for  extensive  concurrency.  The  program  suffered  numerous 
setbacks,  and  the  performance  requirements  w'ere  periodically  relaxed  until  they 
coincided  with  technical  realities.  The  program  suffered  from  high  R&D  and  unit 
cost  overruns  and  was  very  late  relative  to  the  initial  schedule.  VMiile  it  lacked  the  full 
performance  sought  in  the  original  requirements,  it  is  regarded  as  an  important  ex¬ 
pansion  to  force  capability. 


FlOO-PW-220  AND  FllO-GE-IOO  ALTERNATE  FIGHTER  ENGINES  (AFE) 

The  AFE  development  involved  two  parallel  development  efforts  that  ultimately  pro¬ 
vided  the  basis  for  the  "Great  Engine  War,"  a  series  of  annual  production  competi¬ 
tions  between  the  FllO  and  FlOO-PW-220  engines.  To  remedy  serious  operability 
and  supportabilitv'  problems  with  the  FIOO-PW- 100/200  engines  that  it  used  in  its 
F-15  and  F-16  aircraft,  the  Air  Force  pushed  their  manufacturer,  Pratt  and  Whitney 
(P&W),  to  develop  several  significant  modifications.  To  improve  its  leverage  with 
P&W,  the  Air  Force  also  supported  a  small-scale  effort  by  General  Electric  (GE)  to 
derive  a  fighter  engine  from  the  engine  it  had  developed  for  the  B-1  bomber.  As  the 
GE  effort  came  to  fruition,  yielding  the  FllO.  support  for  a  production  competition 
between  this  new  engine  and  a  P&W  engine  grew  in  the  Air  Force.  P&W  eventually 
combined  a  set  of  the  modifications  it  was  developing  into  a  configuration  that 
became  the  FlOO-PW-220,  allowing  the  competition  to  occur. 
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As  suggested  by  the  description  above,  this  development  was  very'  dynamic.  The 
Propulsion  SPO,  which  was  in  place  before  these  developments  started  and  was 
staffed  by  an  experienced  and  technically  sophi.sticatcd  team,  managed  the  devel¬ 
opments  skillfully,  responding  effectively  to  growing  support  for  formal  competition 
and  coordinating  the  parallel  developments  so  that  effective  competition  would  ul¬ 
timately  be  possible.  The  SPO  took  deliberate  actions  to  limit  risk  by  using  as  much 
proven  technology  as  possible  in  the  new  engines,  imposing  demanding  perfor¬ 
mance  warranties  to  ensure  realization  of  its  development  goals,  using  its  technical 
capabilities  to  monitor  both  developments  closely  on  a  day-to-day  basis. 

In  the  end,  the  Air  Force  achieved  its  principal  development  goals  with  approxi¬ 
mately  the  cost  and  schedule  expected.  The  new  GE  engine  forced  P&vv  to  address 
major  problems  in  its  FlOO  engine.  Both  new  engines  embodied  new  reliability  engi¬ 
neering  that  dramatically  im^jioved  the  durability  of  the  engines  the  Air  Force  used  in 
its  F-15  and  F-16  aircraft.  Both  engines  effectively  eliminated  the  stall  stagnation 
operating  problem  that  had  presented  safety-  problems  in  these  aircraft.  The  exis¬ 
tence  of  two  engines  supported  an  ongoing  production  competition  that  improved 
the  Air  Force's  leverage  with  both  contractors.  The  developments  went  .so  well  that 
the  Air  Force  formalized  the  implicit  competitive  approach  it  used  in  thetn  to  de¬ 
velop  the  engines  that  replaced  the  FlOO-PVV-220  and  FI  10. 


GLOBAL  POSITIONING  SYSTEM  (GPS) 

The  GPS  is  a  space-based,  radio  positioning  and  navigation  system  that  is  designed 
to  pro\dde  extremely  accurate  three-dimensional  position  and  velocity'  information, 
together  w'ith  system  time,  to  suitably  equipped  users.  It  is  composed  of  a  constella¬ 
tion  of  satellites  that  broadcast  signals  that  are  interpreted  by  autonomous  receivers; 
the  satellites  are  supported  in  their  operation  by  a  ground-based  control  segment. 
The  GPS  consists  of  three  major  segments:  space,  control,  and  user. 

The  GPS  has  its  roots  in  two  distinct  parts  of  the  rriilitary'  operational  community:  the 
Navy,  which  traditionally  assumed  responsibility  for  the  developmeni  and  imple¬ 
mentation  of  aids  to  marine  navigation,  and  the  Air  Force,  which  became  interested 
in  precise  na\ng,'.tion  because  of  the  lack  of  capability  to  bomb  targets  with  sufficient 
precision  during  the  Vietnam  War.  These  respective  interests  spawned  separate  pro¬ 
grams  in  the  late  1960s  directed  toward  developing  satellite-based  radio-navigation 
capabilities.  These  programs  were  eventually  merged  into  the  GPS  program,  to  ad¬ 
dress  both  needs  u-ith  greater  efficiency. 

The  management  strategy  for  full-scale  development  of  the  Block  I  spacecraft  was 
probably  less  important  to  the  success  of  the  development  than  was  the  manage¬ 
ment  strategy'  for  dem/val.  For  the  dem/val,  the  SPO  chose  to  proceed  with  deploy¬ 
ment  of  a  series  of  experimental  satellites,  learning  progressively  through  expeiience. 
This  trial  and  error  approach  was  possible  in  part  because  of  the  low  visibility  of  the 
program,  a  circumstance  that  generally  allow-ed  technical  imperatives  to  take 
precedence  over  more  political  or  bureaucratic  imperatives. 

FSD  was  structured  to  pursue  only  modest  advances  in  satellite  design  beyond  that 
of  the  dem.N’al  phase.  This  was  less  true  for  user  equipment,  for  wlrich  FSD  involved 
advancing  from  large  experimental  sets  on  pallets  to  much  more  modular  equipment 
iniegrated  in  operational  configurations  into  a  variety  of  combat  platforms.  The  ba¬ 
sic  management  strategy  with  regard  to  the  spacecraft  can  be  described  this  way: 
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beat  down  the  technical  risk  to  low  levels  by  experimentation  and  set  only  modest 
goals  for  technological  advance  during  FSD  (this  was  true  even  before  the  budget 
shortfall  necessitated  a  less  ambitious  set  of  performance  specifications  for  FSD). 

Demonstrating  enabling  technology  for  the  GPS  satellite  in  the  dom/val  phase  seems 
to  have  contributed  substantially  to  the  relatively  smooth  progress  of  I'SD.  There 
were  few  technological  surprises.  The  surprise  with  the  greatest  consequences  for 
the  development  program  as  a  whole  (though  not  for  the  spacecraft  design)  was  the 
Challe  nger  accident,  which  revealed  the  weakness  of  relying  exclusively  on  the  shut¬ 
tle  for  transporting  GPS  satellites  to  orbit. 

This  case  suggests  that  the  distinction  between  dem/val  and  full-scale  development 
is  somewhat  ambiguous  and  circumstantial:  the  GPS  satellite  full-scale  engineering 
development  took  place  in  largo  part  during  dem/val.  In  this  case,  the  transition 
from  proof-of-principlc  experiments  to  operational  capability  proved  to  be  rather 
gradual  and  incremental. 

The  GPS  program  resulted  in  deployment  of  a  system  that  is  generally  regarded  as 
having  met  cost,  schedule,  and  performance  expectations.  Though  there  were  in¬ 
stances  of  cost  overrun,  schedule  slipp.age,  and  modification  of  performance  specil'i- 
entions,  itiese  did  not  fundnmcntallv  alter  the  timeliness  or  util'iv  of  tiie  svstem. 


F-ie/MULTINATIONAL  STAGED  IMPROVEMENT  PROGRAM  (MSIP) 

As  it  began  to  field  its  first  F-16  A/Bs,  the  Air  Force  anticipated  the  need  to  upgrade 
the  aircraft  systematically  over  its  lifetime  as  the  threat  changed  and  new  technical 
capabilities  became  available.  The  Air  Force  worked  with  General  Dynamics  (GD), 
the  F-16  developer,  to  devise  a  formal  mechanism  for  developing  new  versions  of  the 
aircraft.  The  MSIP  was  the  result.  It  ultimately  provided  a  mechanism  for  improving 
new  F-16S  in  a  series  of  stages,  customizing  models  for  each  country  that  bought  the 
aircraft,  upgrading  existing  U.S.  and  foreign  aircraft  to  take  advantage  of  new  techni¬ 
cal  capabilities,  and  developing  derivatives  of  the  F-16  for  specialized  missions. 

The  F-16  SPO  used  time  tested  methods  for  developing  the  F-16  .‘\/B  to  coordinate 
the  testing  and  integration  of  a  large  number  of  subsystems  in  parallel  with  one  an¬ 
other  and  with  'he  formulation  of  complete  new  configurations  for  the  F-16  itself. 
Underlying  multi-year  production  contracts  limited  risk  by  stabilizing  demand 
schedules  for  the  U.S.  aircraft.  The  major  residual  risk  lay  in  subsystems  that  would 
not  perform  as  expected  or  would  not  be  ready  when  promised.  The  MSIP  dealt  with 
this  problem  by  (a)  seeking  subsystems  based  on  well  developed  technologies,  (b) 
coordinating  test  and  integration  activities  with  subsystem  developers  early,  (c) 
preparing  contingencies  to  substitute  available  systems  for  those  not  yet  available  on 
new  production  aircraft  and  to  retrofit  new  subsystems  as  they  became  available, 
and  (d)  using  an  experienced  and  well-trained  staff  to  manage  inevitable  surprises  as 
they  arose.  Such  activities  were  possible  only  because  the  F-16  SPO  had  great  flexi¬ 
bility  and  maintained  close,  mutually  satisfying  working  relationships  with  GD,  in 
particular,  and  most  of  the  other  organizations  involved  in  developing  and  testing 
subsystems. 

Most  observers  view  the  MSIP  as  a  great  success.  Difficulties  in  the  development  of 
subsystems  delayed  the  operational  availability  of  new  capabilities  in  the  F-16  fleet, 
but  the  mechanism  devised  to  maintain  the  preplanned  production  schedule  and 
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bring  new  capabilities  on-iinc  as  they  did  become  available  worked  svell.  Methods 
based  on  the  MSIP  also  allowed  rapid  and  confident  development  oi ..  wide  variety  of 
F-16  variants  tn  match  the  many  missions  that  this  multi-role  fighter  now  serves  in 
the  United  States  and  elsewliere. 

JOINT  SURVEILLANCE  TARGET  ATTACK  RADAR  SYSTEM  (JOINT  STARS) 

The  Joint  STARS  is  an  aircraft-mounted,  radar-based  system  that  can  detect  mobile 
armored  forces.  Joint  STARS  facilitates  the  accurate  attack  of  enemy  forces  by  pro¬ 
viding  position  information  on  enemy  targets  to  direct  attack  aircraft,  artillery,  and 
standoff  missiles.  Additionally,  Joint  STARS  can  be  used  to  gather  wide  area  surveil¬ 
lance  information  on  the  disposition  of  enemy  ground  forces.  The  Joint  STARS  pro¬ 
gram  was  initiated  by  OSD  to  consolidate  separate  Air  Force  and  Army  programs  di¬ 
rected  toward  accomplishing  these  targeting  and  suiweillance  object"  ves. 

For  a  number  of  years  before  loint  STARS,  both  government  and  indusitA'  liad  con¬ 
ducted  demonstrations  of  enabling  technologies  for  long-range  location  and  tracking 
of  arniored  v'diiclcs.  The  Army  laboratory  at  Fort  Monmouth.  New  icrscy,  and  the 
Air  Force  Rome  .Air  Development  Cciiter  in  New  York  (through  tlie  work  of  Lincoln 
l.aboratnry)  liad  each  denmnstrated  not  only  t!.e  laboratem'  fcasiMiiy  of  various  en¬ 
abling  technologies,  but  had  conducted  fieid  demonstrations  to  prove  that  these 
technologies  could  be  usefully  integrated  into  aircraft.  The  enabling  technologies 
were  moving  target  indicator  (MTl)  radar  to  detect  slow  moving  ground  targets,  fixed 
target  indicator  (FTI)  to  identify  and  classify  fixed  targets,  signal  processing  lo  allow  a 
single  antenna  to  serve  both  Unctions,  data  links  to  pass  information,  weapon  in¬ 
terface  units,  and  the  'ike. 

Each  service  addressed  the  operational  objective  of  defeating  enemy  concentrations 
of  heavy  forces,  but  they  had  different  focuses  for  their  efforts;  the  Army  was  most  in¬ 
terested  in  striking  first-echelon  enemy  armor  (relatively  near  the  forward  line  of  bat¬ 
tle]  with  artillery  and  the  multiple  launch  rocket  system  (MLRS).  while  the  Air  Force 
was  most  interested  in  striking  second-echelon  enemy  armor  with  air-io-surface 
weapons.  .Although  technical  concepts  for  locating  and  destroying  enemy  armor  dif¬ 
fered  in  the  candidate  systems,  both  incorporated  airborne  radars 

The  management  strategy  for  Joint  STARS  was  not  so  much  the  product  of  central¬ 
ized  planning  as  it  was  the  result  of  a  series  of  negotiations  among  tlic  Air  Force,  the 
Army,  and  OSD.  The  basic  program/priced  options  approach  reduced  the  risk  that 
full-scale  development  and  production  would  call  for  greater  pcrforn.ancc  than  the 
budget  could  support.  The  choice  of  a  single,  large  plaiform  reduced  a  r  Jt  of  techno¬ 
logical  risks  that  might  not  have  been  easily  managed;  those  of  engineering  a  multi- 
mode  radar  to  fit  into  airframes  with  little  margin  for  error. 

The  formation  of  the  joint  program,  though  difficult,  accomplished  several  important 
functions:  it  began  the  process  of  reconciling  Air  Force  and  Army  views  on  long- 
range  surveillance  needs  for  Airland  Battle;  it  provided  a  forum  for  the  scn.'icer.,  OSD, 
and  the  Congress  to  argue  about  what  lomt  STARS  should  be,  particularly  during  fne 
platform  decision  and  requirements  scrub.  The  formation  of  the  joint  program,  and 
tlic  preparation  for  FSD.  served  as  a  focal  point  for  debate  over  the  desirability  of  the 
system.  The  program  office  pci  formed  its  management  hinctions  rcasonablv  well  as 
this  debase  proceeded.  Tiiough  snnv  of  those  intcrvie'.'ed  complained  about  un- 
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even  qualifications  and  relatively  rapid  turnover  of  personnel,  the  program  office 
nevertheless  managed  risks  with  a  fair  degree  of  success 

The  capacity  of  the  program  office  and  contractors  to  manage  risk  during  fuil-scale 
development  was  largely  determined  by  the  ways  in  which  they  managed  risks  before 
full-scale  dpvelopm.cnt.  Though  the  development  took  several  years  longer  than  was 
originally  hoped,  and  cost  30  percent  more  than  originally  planned,  it  nonetheless 
generated  a  system  that  has  proven  useful  in  an  operational  environment.  This  was 
possible  in  large  part  because  of  skillfully  executed  technological  and  proof-of-prin- 
ciple  demonstrations  preceding  fSD.  which  reduced  the  technical  risks  of  conduct¬ 
ing  the  Joint  STAHS  program. 

The  Joint  STAR.S  program  is  not  yet  complete— production  is  only  now  under  way. 
The  developmental  system  was  deployed  during  the  Persian  Gulf  War,  and  valuable 
operating  experience  gained  has  been  reflected  back  into  the  development  program. 
The  !oim  STARS  program  has  experienced  substantial  delays,  cost  growth,  and  per¬ 
formance  changes,  due  in  part  to  a  concept  for  operation  of  the  system  that  has 
evolved  coniiniioLisly. 


FINDINGS  ACROSS  T1  IF  CASE  STUDIES 

At  the  beginning  of  this  chapter,  wc  listed  nine  lactors  important  in  understanding 
the  management  of  risk  in  weapon-system  development  programs.  The  case  studies, 
taken  jointly,  confirm  the  importance  of  these  factors.  In  this  section,  we  summarize 
this  experience.  At  the  end  of  the  chapter.  Table  3  provides  thumbnail  comments 
about  each  factor  for  each  case.  Those  interested  in  more  detailed  treatment  of  par¬ 
ticular  programs  should  examine  die  individual  case  studies. 

State  of  the  Underlying  Technology 

To  minimize  risks  during  development,  acquisition  policy  has  devoted  considerable 
attention  to  assuring  that  the  technology'  required  for  a  new  system  is  well  in  hand 
before  large  commitments  arc  made  to  that  system.  Emphasis  on  the  dem/val  phase 
of  a  program,  strong  pressures  to  make  use  of  prototypes,  and  advocacy  of  \ngorous 
exploratory'  and  advanced  dcvelopi..ent  programs  are  all  intended  to  assure  'hat  sys¬ 
tems  enter  full-scale  development  either  with  their  icclinologies  well  in  hand  or  with 
problem  areas  clearly  identified  ..iid  plans  for  dealing  with  them  in  place. 

Despite  these  poiicics,  tlie  ted. oology  underlying  several  of  the  programs  hr  our 
sample  was  clearly  not  fully  developed  and  ’’Itimately  caused  substantial  diifi  .ulties 
for  liicsc  programs. 

The  Quality  and  .Stability  of  the  Requirement  for  the  System 

The  DoD  5000  series  directives  and  Office  of  .Management  and  Budget  (0MB)  direc¬ 
tive  A- 109  place  considciablc  emphasis  on  beginning  the  acquisition  process  With  a 
well-developed  and  specified  statement  of  needs.  If  the  requirement  is  unstable, 
there  is  an  ini  reased  probability  that  the  system  will  come  to  be  viewed  as  lacking 
appropriate  performance  attributes,  or  that  cost  increases  and  schedule  slippage  will 
occur  as  the  program  scrambles  to  adjust  to  new  perceptions  of  operational  needs. 
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In  tlie  past,  soitig  systems  were  started  before  needs  were  well-understood  and  had 
to  be  modified  as  the  understanding  of  threat  and  doctrine  evolved.  In  other  cases, 
the  requirement  was  the  product  of  adversarial  bargaining  among  constituencies 
with  a  stake  in  the  approval  of  the  system,  and  it  changed  as  the  bargaining  pro¬ 
ceeded. 

With  the  exception  of  Joint  STARS,  the  requirements  for  the  systems  that  we  exam¬ 
ined  were  well-specified  and  stable.  As  always,  with  hindsight,  one  might  have 
specified  a  requirement  differently;  our  finding  is  that,  with  the  single  exception,  in¬ 
stability  of  the  requirement  did  not  cause  great  development  problems. 

The  Quality'  and  Continuity  of  Experience  of  the  SPO  Personnel 

As  we  have  noted,  the  management  of  risk  requires  considerable  expertise  to  identify 
where  risks  are  and  to  devise  progi-im  strategies  that  contain  and  ameliorate  that 
risk,  This  in  turn  requires  personnel  with  training  and  experience— particularly  ex¬ 
perience  with  the  technologies  germane  to  the  system  and  the  specific  development 
activities  leading  to  the  full-scale  development.  Absent  such  staff  and  experience, 
the  risks  to  the  Air  I'orce  arc  increased  because  of  the  reduced  ability  of  tlie  SPO  to 
anticipate  and  respond  to  adverse  events  affccling  program  outcomes. 

We  found  no  procedures  that  dealt  well  with  risk  in  the  absence  of  qualified  people  in 
the  program  office.  In  our  sample,  there  was  considerable  variability  in  the  manner 
in  which  SPOs  were  staffed  and  the  quality  of  the  SPO  leadership.  These  had  marked 
effects  on  the  program  outcomes. 

Features  of  Management  Strategy 

There  are  a  variety  of  features  of  management  strategy  that  have  important  impacts 
on  risk.  Most  prominent  in  our  sample  and  in  policy  debate  is  the  level  of  concur¬ 
rency  among  technological  developmeitt,  full-scale  development,  and  production. 
The  higher  the  level  of  concurrency  in  these  activities,  all  other  things  equal,  the 
greater  the  risk  of  schedule  slip,  performance  shortfalls,  or  cost  increases — or  all 
three. 

Although  concurrency  is  tlie  most  prominent  example  of  a  management  strategy', 
there  are  others.  Several  of  tlic  programs  carried  out  parallel  developments  to  lower 
risks;  one  carried  a  second  source  to  maintain  cost  discipline  and  minimize  the  pos¬ 
sibilities  of  cost  escalation. 

Management  strategies  were  clearly  articulated  by  the  senior  staff  of  the  program  of¬ 
fices.  The  problems  inherent  in  high  levels  of  concurrency  are  so  widely  recognized 
that  they  are  always  an  extensively  debated  issue  in  program  planning;  the  programs 
examined  are  no  exception.  The  problem,  at  least  as  shown  in  our  case  studies,  is 
obtaining  and  using  good  information  on  the  program-specific  risks  associated  with 
concurrency.  It  appears  that  the  variety  of  actors  with  interests  in  programs  such  as 
the  B-lB,  lANTIRN,  and  AM^1.■\.^M  held  widely  d.ffcring  perceptions  on  the  nature 
and  levels  of  these  risks. 
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Contractual  Terms  and  Conditions 

Wlien  this  study  started,  there  had  been  significant  complaintb  by  aerospace  contrac¬ 
tors  about  the  use  of  fixed-price  contracts  for  research  and  development,  These  con¬ 
tractors  and  their  allies  claimed  that  such  contracts,  particularly  when  awarded  in 
sharply  contested  competitions,  placed  too  much  risk  on  the  contractor,  Several 
contractors  established  policies  of  refusing  to  accept  such  contracts.  The  most 
prominent  adverse  effect  cited  was  die  possible  erosion  of  the  industrial  base  as 
firms  chose  to  leave  the  market,  but  It  is  also  clear  that  contract  terms  that  cause  one 
or  the  other  party  to  bear  major  financial  responsibility  for  cost,  performance,  or 
schedule  deficiencies  will  shape  program  decisions  and  the  relative  power  of  the 
parties  to  the  agreement. 

Virtually  all  the  major  contracts  in  our  sample  were  fixed-price  contracts.  However, 
there  was  a  rich  variety  in  their  terms  and  conditions,  the  warranties  required,  die 
indemnities  provided,  the  process  by  which  engineering  changes  were  agreed  to,  and 
the  ceilings  and  share  r-^tes  that  were  imposed.  Our  sense  was  that  the  high-level 
policy  debates  over  contract  terms  are  largely  irrelevant  to  what  actually  happens  be¬ 
cause  of  the  complexity  of  the  contractual  agreements,  .\lthough  our  studic.s  were 
too  limited  to  investigate  contracts  tlioroughly,  it  seems  clear  that  an  appreciation  of 
the  nature  and  level  of  risks  strongly  informed  the  actual  bargains  struck. 

Requirements  for  Intra-Governmental  or  Intra-Service  Agreement 

Three  of  the  seven  programs  had  joint  sponsorship  by  the  Air  Force  and  the  ,\avy,  al¬ 
though  the  Na\7  played  but  a  small  role  In  the  Alternate  Fighter  Engine  develop¬ 
ment.  Most  involved  significant  cooperation  with  other  development  efforts,  either 
because  they  relied  on  components  developed  for  other  systems  or  because  they 
were  to  be  integrated  with  some  other  system.  One  program,  the  MSIP,  was  predom¬ 
inately  concerned  with  coordinating  the  efforts  of  different  program  offices  and  con¬ 
tractors. 

On  its  face,  "jointness"  in  the  conduct  of  a  development  effort  seems  likely  to  in¬ 
crease  risk  because  of  the  differences  in  perspectives  on  requirements  as  well  as  pri¬ 
ority  to  the  services  or  SPOs.  Our  cases  include  examples  of  both  highly  successful 
joint  efforts  (GPS)  and  less  successful  ones  (loint  STARS).  The  MSIP  program  had 
worked  out  a  quite  routinized  and  apparently  successful  procedure  for  dealing  with 
the  coordination  problems. 


The  Use  of  Competition  Throughout  the  Development  Program 

The  contractors  for  FSD  in  the  sample  were  selected  in  a  variety  of  wax....  In  most 
cases,  some  form  of  competition  was  involved.  But  the  situations  were  complex.  For 
example,  there  are  only  two  major  aircraft  engine  manufacturers  in  the  United 
States.  In  the  AFE  program,  development  of  the  engines  proceeded  in  parallel 
through  the  early  phases  of  what  amounted  to  full-scale  development.  Production 
contracts  that  included  development  work  were  awarded  to  both,  but  the  negotiated 
terms  were  the  result  of  playing  one  contractor  off  against  the  other.  ,.\  competition 
was  held  for  the  dem/val  phase  of  the  GPS  and  the  winner  was  carried  through  the 
initial  production  contract.  A  new  competition  then  resulted  in  changing  contrac¬ 
tors,  something  that  appears  possible  in  spacecraft  but  less  so  in  aircraft  or  missile 
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systems,  In  the  AMRAAM  program,  a  second  source  was  carried  tlirough  the  devel* 
opment  and  production  phases.  In  one  case,  the  B-IB,  there  was  no  competition,  in 
part  as  a  result  of  the  contractors'  participation  in  tlie  prior  B-IA  program  and  in  part 
because  it  was  felt  there  was  insufficient  time  or  money  to  afford  the  competition, 

The  presence  of  competition  has  obvious  effects  upon  program  risks,  most  of  them 
positive,  A  spirited  compedtion  for  a  development  contract  will  often  reveal  in¬ 
formation  about  technical  and  production  risks  in  the  program  because  of  the 
distinctive  approaches  that  the  competitors  take.  Continuing  competition  during 
development  provides  both  a  hedge  against  the  performance  failure  of  a  particular 
contractor  (and  its  technical  approach)  and  a  tangible  motivation  to  the  competitors 
to  perform  well.  However,  competition  coupled  with  lack  of  realism  on  the  part  of 
the  program  office  and  higher-ups  can  lead  to  overly  optimistic  promises  regarding 
performance,  schedule,  and  costs,  which  may  subsequently  runstrain  program  man¬ 
agement  decisions  in  undesirable  ways.  To  the  extent  that  those  promises  become  the 
criteria  by  which  the  overall  performance  of  the  program  is  judged,  competition  may 
actually  increase  the  risks  inherent  in  the  program. 

The  case  studies  make  it  clear  that  the  nature  of  the  competition  influences  a  pro¬ 
gram  outcome.  No  senior  program  official  with  whom  we  talked  wanmd  to  conduct 
a  program  without  having  a  credible  threat  of  competition;  similarly,  no  contractor  is 
anxious  to  be  totally  dependent  on  a  single  supplier.  The  AFE  case  provides  a  striking 
example  of  what  might  be  called  managed  competition  that  only  in  its  final  stages 
involved  a  formal  competition  among  sources.  At  the  same  time,  tlie  situations  ex¬ 
amined  were  so  rich  and  varied  that  it  is  difficult  to  see  how  policymakers  can  pre¬ 
scribe  a  standard  means  for  creating  competition. 

The  Nature  of  the  Contractors 

Outcomes  are  clearly  influenced  by  the  experience,  the  competitive  position,  the  fi¬ 
nancial  strength,  and  the  level  of  business  activity  of  the  contractor.  In  several  cases, 
problems  were  faced  (and  risks  increased)  because  other  programs  were  making  de¬ 
mands  on  a  limited  design  and  development  staff.  In  others,  firms  devoted  addi¬ 
tional  resources  beyond  those  on  contract  (presumably  reducing  risks)  because  they 
were  seeking  entry  into  a  new  market. 

In  a  couple  of  cases,  contractors  "boughi  into  a  program,”  proposing  prices  that  were 
lower  than  the  costs  they  expected  to  incur  to  enter  a  new  market  or  solidify  market 
share.  They  undoubtedly  expected  to  make  this  back  either  during  subsequent  pro¬ 
duction  contracts  or  because  of  an  enhanced  future  market  position.  However,  if 
subsequent  events  suggest  that  production  will  not  be  there  or  the  market  is  less  im¬ 
portant,  the  prospect  of  losses  (coupled  in  some  cases  with  tenuous  financial  posi¬ 
tions)  may  constrain  responses  to  untoward  technical  and  operational  events. 

In  general,  in  the  cases  we  examined,  it  appeared  that  the  Air  Force  had  a  good  un¬ 
derstanding  of  the  nature  of  the  contractors  when  they  entered  development. 
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The  Nature  of  Constraints  External  to  the  Program  Created  by  Congress,  the 
DoD,  or  Headquarters  USAF 

DoD  directives  and  regulations  generally  emphasize  risks  associated  with  features 
internal  ro  a  program,  The  development  process  is  intended  to  deal  with  technologi¬ 
cal  and  managerial  uncenaintieL.  A  few  policies,  such  as  those  governing  price  esca¬ 
lators  and  indemnitication  clauses,  deal  with  well-deuned  problems  external  to  the 
program.  But,  in  fact,  a  great  deal  of  risk  management  in  the  programs  we  examined 
dealt  with  risks  external  to  the  program.  In  particular,  program  managers  regularly 
had  to  manage  responses  to  changing  budget  levels,  the  imposition  of  spending  caps, 
special  testing  and  reporting  requirements,  and  newly  imposed  schedules.  The'^  had 
to  be  prepared  to  deal  with  events  that  were  highly  unpredictable  and  full  of  appar¬ 
ent  arbitrariness. 

Implicitly,  most  managers  felt  they  were  dealing  with  the  risk  of  cancellation  or  sub¬ 
stantial  cutbacks  in  their  programs.  For  example,  whate  ’er  their  formal  justification, 
many  of  their  actions — the  early  commitment  to  production  or  a  preference  for 
multi-year  contracts — can  be  seen  as  a  means  of  lessening  the  risk  of  cancellation  or 
cutback.  Although  we  have  no  clear  means  of  measuring  it,  our  interviews  and  re¬ 
views  of  program  files  suggest  that  much  of  the  most  senior  managers'  time  was 
spent  in  managing  risks  to  the  program  inherent  in  these  external  events.  This  was 
particularly  true  for  high-visibility  programs  related  to  core  missions  of  the  Air  Force. 

SUMMARY  OF  FACTORS  AFFECTING  RISK  IN  SEVEN  PROGRAMS 

Our  case  studies  describe  each  program  in  terms  of  each  of  these  factors  and,  as  we 
repeatedly  note,  they  should  be  read  for  a  fuller  story,  However,  we  summarize  o  ur 
findings  in  Table  3.  This  summary  is  used  in  the  analysis  of  risk  management  con¬ 
tained  in  the  next  chapter. 


Factors  Affecting  Risk  in  Case  Studies 
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DETERMINANTS  OF  LEVEL  AND  MANAGEMENT  OF  RISKS 


We  found  it  useful  to  group  the  seven  programs  we  examined  into  three  categories. 
Three  programs,  the  B-IB,  AMRAAM,  and  LAN'TIRN,  seemed  to  us  to  experience 
substantial  acquisition  difficulties  during  full-scale  development.  (Because  the  B-lB 
was  a  fully  concurrent  program,  the  difficulties  extended  through  the  production  as 
well.)  Three,  the  Alternate  Fighter  Engine  (AFE),  Global  Positioning  System  (GPS), 
and  Multinational  Staged  Iriprovcmeni  Program  for  the  F-16  (V1S1P),  proceeded 
fairly  smoothly,  The  final  program,  loint  STARS,  experienced  modest  difficulties.  In 
addition,  of  the  programs  examined,  it  turned  out  to  be  the  furthest  from  completion 
at  the  time  of  our  research. 

It  is  important  to  emphasize  that  we  are  not  making  judgments  about  the  ultimate 
performance  of  these  developments.  The  Air  Force  and  the  nation  may  well  have 
obtained  crucial  capabilities  at  reasonable  prices  and  in  a  timely  fashion,  even  in 
those  programs  experiencing  acquisition  difficulties— and  may  not  have  done  so  in 
programs  that  proceeded  smoothly.  Acquisition  difficulties  are,  to  a  large  extent, 
measured  by  the  degree  to  which  planned  performance,  schedule,  and  cost  objec¬ 
tives  are  met.  It  may  be  that  these  planned  outcomes  are  unnecessarily  ambitious.  If 
the  program  had  been  planned  with  lower  levels  of  performance  or  a  less  tight 
schedule,  it  would  have  been  deemed  a  model  of  acquisition  management.  This, 
however,  is  a  judgment  considerably  beyond  the  reach  of  this  research  report. 

The  three  programs  categorized  as  experiencing  significant  acquisition  difficulties 
share  a  number  of  important  characteristics: 

1.  Each  was  viewed  as  central  to  key  Air  Force  missions. 

2.  All  possessed  demanding  cost,  schedule,  and  performance  objectives  that  resulted 
from  uvo  important  pressures:  (1)  strong  demands  from  operational  commands; 
and  (2)  the  need  to  sell  the  program  to  authorities  outside  the  Air  Force, 

3.  All  included  a  high  degree  of  concurrency  between  development  and  production. 

4.  Each  was  subjected  to  important  constraints  (funding  caps,  mandated  schedules, 
etc.)  that  limited  the  capacity  of  the  program  office  to  manage  the  program. 

5.  In  each,  there  was  lack  of  expertise,  personnel,  and  continuity  in  the  SPO  at  some 
key  points  in  the  development  process. 

6.  For  a  variety'  of  reasons,  top-level  management  oversight  in  the  Air  Force  and  DoD 
discounted  lower-level  assessments  of  the  level  of  risk. 
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7.  For  each,  there  were  important  "holes"  in  the  technology  base  that  were  recog¬ 
nized  but  de-emphasized  in  the  advocacy  for  the  program. 

8.  In  two  of  the  three  programs,  the  contractor  bought  into  the  program. 

In  contrast,  the  programs  that  proceeded  fairly  smoothly  from  an  acquisition  man¬ 
agement  viewpoint  were  characterized  by: 

1.  Technologies  that  were  well  in  hand  or  handled  effectively  in  earlier  phases  of  the 
development  process. 

2.  Modest  levels  of  concurrency. 

3.  Flexibility  to  make  schedule,  cost,  and  performance  tradeoffs. 

4.  Strong  and  technically  qualified  program  offices  that  possessed  significant  ele¬ 
ments  of  staff  continuity. 

5.  Relatively  little  external  oversight. 

Just  as  significant,  there  were  a  number  of  features  of  these  programs  that  do  not  dz/- 
/er  systematically  across  these  programs,  features  that  are  important  subjects  of  ac¬ 
quisition  management  policy  for  the  development  phase.  In  particular: 

1.  There  were  no  systematic  differences  in  the  types  of  contracts  used.  All  used 
fixed-price  contracts  during  development:  several  used  firm  fixed-price  contracts. 
But  the  terms  of  these  contracts  varied  widely  in  areas,  such  as  the  share  line, 
ceiling,  and  warranty. 

2.  Most  had  a  consistent  requirement  through  development,  although  Joint  STARS 
had  requirement  instability  due  to  difficulties  in  resolving  Air  Force  and  Army 
differences  concerning  the  operational  concept  for  using  the  system. 

3.  There  were  a  spectrum  of  source  selection  procedures  ranging  from  sole  source  to 
sustained  competition,  but  they  did  not  seem  consistently  related  to  acquisition 
management  outcomes. 

4.  The  developments  did  not  differ  in  the  timing  of  when  technological  problems 
that  faced  developers  in  FSD  were  knowm.  In  all  cases,  these  problems  were 
identified  either  very  early  in  FSD  or  before. 

5.  All  (except  the  B-IB)  dealt  with  significant  externally  imposed  budget  changes  in 
the  course  of  their  developments. 

WTiat  are  the  implications  of  these  observ’ations  for  the  management  of  risk?  Based 
on  our  case  studies,  we  conclude  tliat  the  key  ingredients  of  successful  risk  manage¬ 
ment  are  qualified  technical  staff,  possessing  sufficient  flexibility  (or  slack)  to  re¬ 
spond  to  both  expected  or  unanticipated  difficulties  in  the  program,  coupled  with 
oversight  that  is  qualified  and  that  fcrees  consideration  of  program  issues  that  are 
beyond  the  purvnew  of  the  program  managers.  This  is  hardly  a  novel  finding. 
Current  policy  directives  and  pronouncements  are  certainly  consistent  with  the 
ideas,  although  one  might  argue  that  the  exercise  of  judgment  and  the  flexibility  to 
make  decisions  may  be  inhibited  in  the  environment  created  by  a  wide  array  of  regu¬ 
lations,  directives,  and  mandates  imposed  by  Air  Force  Headquarters,  DoD,  or  the 
Congress. 


If  this  relatively  straightforward  requirement  for  good  risk  management  is  widely  un¬ 
derstood,  why  do  programs  experience  acquisition  problems?  Our  cases  suggest  that 
the  reasons  are  systemic — related  to  the  entire  array  of  cultural  norms  and  incentives 
drat  govern  the  actimties  of  dre  Air  Force  and  the  DoD.  The  key  problem  is  die 
strongly  felt  need  to  promise  the  very  edge  of  plausible  outcomes—in  other  words,  to 
overpromise.  This  need  is  inherent  in  the  decisionmaking  environment  within 
which  new  systems  need  to  be  sold.  Proponents  of  a  particular  system  or  system 
concept  face  diis  at  the  initiatioir  of  a  program  when  technical  and  operational  ideas 
are  competing  to  become  the  basis  for  design  concepts,  during  contract  competi¬ 
tions  when  designs  are  compedng  with  one  anodrer,  at  headquarters  levels  when  one 
command’s  system  is  compering  with  another’s  for  funding,  and  at  OSD  and 
Congress  where  the  systems  (and  other  resource-consuming  activities)  are  compet¬ 
ing  with  those  of  dre  other  services  or  other  departments  of  the  government.* 

The  pressure  to  overpromise  is  high  when  the  program  is  competing  to  enter  devel¬ 
opment,  but  it  is  also  significant  during  the  course  of  the  program.  If  the  program 
runs  into  technical  or  management  difficulties  or  if  the  perceived  threat  changes 
somewhat,  there  is  every  incentive  to  portray  the  possibilities  of  "getting  w'ell''  in 
terms  as  glowing  as  politically  feasible. 

The  result  of  this  bargaining  process  can  easily  be  overly  ambitious  and  rigid  cost, 
schedule,  and  performance  goals.  In  terms  of  Figure  1,  these  systemic  pressures  are 
forcing  the  outcome  line  O  to  the  right,  with  the  result  that  the  risk  of  significandy 
negative  program  outcomes  are  increased.  These  outcomes  are  often  promised  to 
individuals  with  oversight  responsibility  in  Headquarters  USAF,  OSD,  or  the 
Congress  who  possess  limited  technical  backgrounds  and  thus  may  have  limited  ca¬ 
pability  to  assess  their  reasonableness.  The  promises  may  become  the  basis  for  un- 
realisdc  expectations  on  the  part  of  these  individuals,  which,  if  frustrated,  may  lead 
them  to  impose  external  constraints,  such  as  funding  caps  and  schedule  require¬ 
ments  that  constrain  tlie  responses  of  program  managers  to  the  problems  that  in¬ 
evitably  arise  during  the  course  of  tlieir  programs. 

These  problems  are  exacerbated  by  another  inherent  reality  of  bureaucratic  organi¬ 
zations.  It  is  very  difficult  to  communicate  information  about  uncertainty  and  risk  in 
complex  organizational  en\ironments.  The  complex  (and  often  intuitive]  nature  of 
that  information  is  not  easily  packaged  in  the  formal  reports  and  briefings  that  are 
the  communications  media  of  large  organizations. 

Moreover,  as  we  have  already  noted,  acquisition  management  occurs  in  the  midst  of 
a  large,  often  adversarial  system.  Important  actors  in  that  system  are  not  anxious  to 
dwell  on  risks.  Telling  an  audience  that  there  is  a  15-percent  probability  of  a  particu  ¬ 
lar  performance  deficiency  does  not  help  one’s  case,  particularly  when  to  be  accurate 
it  must  be  accompanied  by  five  minutes  of  qualifications.  There  are  a  number  of  in¬ 
stances,  in  the  cases  we  examined,  where  higher  authorities  appear  to  have  ignored 
or  disagreed  with  assessments  of  risks  or  even  to  have  created  reporting  forums  in 
which  it  was  unlikely  that  a  full  discussion  of  risk  was  likely.  Because  of  the  limited 
nature  of  the  cases,  we  cannot  fairly  assess  the  understanding  and  motivations  of 
these  authorities.  Much  of  our  information  is  taken  from  program  office  records  and 

*VVe  frame  this  incentive  in  terms  of  promises  for  program  outcomes.  However,  the  same  incentives  can 
lead  to  the  lendencv-  to  persuade  decisionmakers  (again  all  up  and  down  the  line)  that  the  threat  is  higli 
and  requires  the  weapon  system  with  the  r>  pe  of  outcomes  that  are  being  promised. 
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interviews  and  thus  disproportionately  represents  the  perspectives  of  tliis  level  of 
the  organization.  Nonetheless,  such  behavior  is  widely  discussed  by  individuals 
throughout  the  DoD  and,  indeed,  in  any  large  organization. 

Vvliy  then  did  our  systems  differ  in  their  apparent  ability  to  effectively  manage  risk 
and  to  move  through  the  acquisition  process?  We  argue  that  those  programs  experi¬ 
encing  difficulty  were  relatively  large  arid  important  to  key  Air  Force  interests.  The 
advocacy  required  to  advance  them  through  the  system  led  to  early  and  often  overly 
optimistic  promises  concerning  tlieir  outcomes.  They  were  so  important  to  the  Air 
Force  that  it  was  willing  in  some  instances  to  agree  to  funding  levels,  schedule  guar¬ 
antees,  atid  funding  caps  that  seem  to  be  unrealistic  and  unvvarranted.  To  the  extent 
that  acquisition  problems  are  measured  simply  by  slippage  from  these  goals,  this 
might  be  a  sufficient  explanation.  But  they  are  not.  The  programs  that  did  compara¬ 
tively  well  also  slipped  in  performance,  schedule,  and  cost,  but  not  to  the  degree  that 
these  systems  did. 

More  significant  from  the  risk  management  perspective  is  that  these  tight  and  well- 
publicized  specification.s  squeeze  the  slack  out  of  the  process  that  we  argue  man¬ 
agers  need  to  manage  risks.  In  the  LAN'TIRN'  program,  parallel  technological  devel¬ 
opments  initially  proposed  to  hedge  against  risks  in  the  system  as  a  whole  and  then 
in  an  important  component  (the  ATR)  were  scrubbed  because  there  were  too  few  re¬ 
sources.  In  the  B-IB,  the  agreed-to  schedule  did  not  allow  the  defensive  avionics  to 
proceed  througli  the  necessary  development  before  production  decisions  were 
made.  More  subtly,  the  decision  to  have  the  Air  Force  manage  the  systems  integra¬ 
tion  for  the  B-IB  (coupled  with  understaffing  in  the  SPO)  meant  that  there  were  lim¬ 
ited  staff  resources  to  deal  with  and  craft  solutions  to  problems  that  arose  during  the 
development  phase.  And  the  high  levels  of  concurrency  in  each  of  the  three  systems 
reduced  the  options  available  for  fixes  because  of  the  constraints  imposed  by  existing 
production  decisions. 

In  contrast,  the  more  managerially  successful  systems  we  examined  did  not  require 
the  same  level  of  promises.  They  tended  to  be  far  less  visible  up  the  management 
line.  The  MSIP  and  the  AFE  both  evolved  out  of  strong  and  highly  organized  SPOs. 
The  GPS  was  planned  by  a  technically  qualified  program  office  that  enjoyed  a  special, 
personal,  and  direct  relationship  with  DDR&E;  this  seems  to  have  been  possible 
because  the  program  was  not  absolutely  front  and  center  on  tlie  Air  Foice's  agenda. 
The  programs  proceeded  with  comparatively  broad  specifications  so  that  their 
managers,  when  confronted  with  problems,  could  make  tradeoffs  they  felt  benefited 
the  entire  system. 

Although  it  may  be  partially  a  function  of  the  authors'  interests,  the  most  important 
parts  of  the  stories  concerning  the  AFE  and  the  GPS  lie  not  in  the  full-scale  develop¬ 
ment  but  in  the  technologv'  and  operational  demonstration  programs  that  preceded 
them.  These  phases  did  not  seem  to  loom  as  large  in  the  less  managerially  successful 
programs,  probably  because  the  need  was  seen  as  so  urgent  that  the  ,''ir  Force 
pushed  ahead  before  these  phases  had  fully  run  their  course.  (In  the  case  of 
LAN’TIRN,  the  advanced  technologies  part  of  the  program  was  shrouded  in  covers  of 
special  access  programs,  which  limited  the  ability  of  anyone  to  know  their  implica¬ 
tions,  and,  in  any  case,  few  if  any  of  the  program  office  personnel  had  experience  in 
these  programs.) 
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The  systemic  nature  of  these  problems  limits  the  ability  of  the  Air  Force  (or  OSD)  to 
craft  policies  that  adequately  deal  with  the  problems  we  describe.  For  example,  the 
finding  that  programs  that  did  well  in  managing  risk  had  relatively  little  top-level 
management  oversight  does  not  lead  to  a  prescription  to  leave  program  offices  alone. 
Indeed,  despite  the  generally  limited  oversight  of  the  GPS,  such  oversight  was  abso¬ 
lutely  crucial  in  shaping  the  definition  of  the  program  in  its  Initial  stages.  Moreover, 
it  is  clear  no  organization  with  large  programs  crucial  to  its  core  mission  will  permit 
such  programs  to  proceed  udthout  oversight.  VVhat  is  needed  is  oversight  that  is  tai¬ 
lored  to  the  needs  of  specific  situations— that  varies  with  the  urgency  of  the  program, 
die  nature  of  its  ultimate  users,  the  size  of  the  effort,  and  so  forth.  This  need  for 
tailored  solutions  extends  to  many  other  areas— choice  concerning  development 
strategy,  the  staffing  of  program  offices,  the  types  of  contractual  arrangements  used, 
and  the  nature  of  competition. 

Rigid  and  binding  policies  will  not  achieve  these  tailored  solutions.  Unfortunately, 
most  of  the  people  in  staffs  above  program  offices  are  limited  to  making,  monitoring, 
or  enforcing  such  policies.  Tailored  solutions  require  program-specific  fact-finding, 
familiarity  with  the  management  and  technological  problems,  a  capacity  to  make 
discretional^-  decisions,  and  considerable  wisdom.  It  also  requires  an  ability  to  make 
and  correct  mistakes.  These  solutions  arc  difficult  to  formulate  and  implement  in 
any  public  agency  subject  to  legislative  and  media  oversight,  but  they  are  even  more 
difficult  in  the  military'  sendees.  In  these  services,  the  normal  personnel  rotation  as¬ 
sociated  vsith  service  career  progression  and  the  comings  and  goings  of  political  ap¬ 
pointees  militate  against  both  the  acquisition  of  expertise  and  the  exercise  of  deeply- 
informed  and  understanding  oversight.  Finally,  the  incentives  driving  program 
advocates  are  so  suong,  so  deeply  ingrained  in  the  organization,  and  so  intimately 
associated  with  other  important  organizational  imperatives,  diat  it  is  difficult  to  see 
how  they  can  be  contained  by  transient  overseers  or  policy  rules. 

We  believe  there  are  some  management  lessons  uncovered  by  our  research  that  de¬ 
serve  further  consideration  We  turn  to  these  in  the  last  chapter.  However,  we  do  not 
believe  that  they  provide  robust  answers  to  the  problems  we  set  out  to  address.  WTiat 
is  imponant  is  that  those  who  oversee  development  acthaties  or  make  policies  that 
govern  such  activities  appreciate  the  nature  of  the  environment  in  which  they  oper¬ 
ate.  The  cases  may  provide  some  of  this  appreciation — and  we  think  they  are  best 
read  if  one  is  not  looking  for  villains  but  rather  seeks  to  understand  how  the  organi¬ 
zational.  political,  and  legal  systems  in  which  these  actisities  take  place  shape  the 
outcomes  that  are  found. 
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CONCLUSIONS 


The  case  studies  that  provide  the  empirical  basis  of  our  report  do  not  suggest  that 
there  are  significant  problems  udth  Air  Force  or  DoD  acquisition  policy.  It  is  Air 
Force  and  DoD  policy  that  programs  should  not  enter  full-scale  development  before 
they  possess  a  well-developed  and  understood  technolog\'  base;  the  case  .studies 
amply  illustrate  the  importance  of  this  policy.  Policy  dictates  that  a  well-developed 
and  agreed-upon  statement  of  mission  need  coupled  with  flexibility  in  definition  of 
ways  to  meet  that  need  is  a  precondition  to  entering  development;  the  case  studies 
illustrate  die  desirability  of  this  policy.  Policy  emphasizes  that  programs  arc  to  be 
managed  and  overseen  by  expert  staff;  the  case  studies  make  it  clear  that  this  is  very 
important  to  the  successful  management  of  risk.  Air  Force  and  DoD  policies  empha¬ 
size  short,  direct  lines  of  communication  between  program  managers  and  authorita¬ 
tive  acquisition  executives;  again,  the  cases  suggest  the  importance  of  such  direct 
links  to  the  quality  of  communication  concerning  risk. 

These  and  many  other  acquisition  policies  seem,  in  concept,  appropriate.  The 
problems  come  in  their  application.  In  the  programs  wc  examined,  acquisition 
problems  most  often  arose  from  the  character  of  the  organizational  system  in  which 
the  policies  are  implemented.  This  system— the  services,  the  DoD  as  a  whole,  and 
the  Congress—operates  in  a  highly  political,  advocacy-oriented  environment. 
Resources  seem  perpetually  to  be  limited  relative  to  the  needs  that  one  or  another 
component  of  the  system  views  as  key  to  its  mission.  In  a  wide  variety  of  forums,  ad¬ 
vocates  for  the  development  of  new'  systems,  or  for  the  continued  acquisition  of 
existing  systems,  vie  for  those  rerources.  In  the  course  of  these  proceedings, 
decisions  are  made  and  bargains  struck  that  sometimes  undermine  the  intent  of  the 
policies. 

In  the  face  of  potential  abridgments  of  policies,  oversight  organizations  often  try  to 
strengthen  these  policies  by  making  them  more  prescriptive.  At  various  times,  the 
military'  setv'ices  have  been  told  that  they  shall  use  prototy  pes  or  sha’l  avoid  concur¬ 
rency,  Fo'ilow'ing  industry'  complaints,  it  became  pr'icv  not  to  use  firm  fixed-price 
contracts  for  development,  hi  the  face  of  stringent  competition  for  funds  and  au¬ 
thority,  these  policies  may  also  seek  to  limit  cost  growth  or  schedule  slippage  by  im¬ 
posing  funding  caps  or  making  further  funding  dependent  on  meeting  a  specified 
schedule.  Similarly,  further  funding  or  authorization  to  enter  into  production  may 
be  made  dependent  on  demonstrating  a  specified  performance  in  a  test  situation. 
These  attempts  to  control  development  efforts  are  understandable  and  may,  taking 
full  national  interests  into  account,  even  be  desirable.  Bu’  it  should  be  understood 
that  they  may  very'  well  limit  the  capabilir/ of  the  program  office  to  manage  the  risks 
inherent  in  the  program. 
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The  keys  to  managing  risk  effectively  appear  to  be  having  qualified  people,  possess¬ 
ing  adequate  "slack"  in  the  program  to  cope  with  adverse  events,  and  being  overseen 
by  people  capable  both  of  understanding  development  issues  and  reflecting  con¬ 
cerns  of  the  larger  policy  environment  on  decisions  concerning  those  adverse  events. 
By  slack  we  mean  the  ability  to  make  cost,  performance,  and  schedule  tradeoffs  in 
dealing  with  those  adverse  events.  Plan.s  and  policies  that  significantly  reduce  the 
slack  available  in  the  program  (e  g.,  by  simultaneously  tightening  cost,  schedule,  and 
performance  imperatives),  possibly  coupled  with  the  failure  to  staff  program  offices 
with  adequate  numbers  of  qualified  staff,  appear  to  be  the  principal  program 
features  that  differentiate  the  programs  with  significant  acquisition  difficulties  from 
those  that  proceeded  relatively  smoothly. 

Policy  is  important.  Bad  policies  are  likely  to  lead  to  bad  outcome.s.  But,  based  on 
these  seven  case  studies,  the  Air  Force  and  the  DoD  seem  to  us  lo  have  the  broad 
policies  about  right.  Problems  arise  when  the  intent  of  those  policies  is  violated, 
This  violation  comes  when  oilier  imperative.s— perhaps  preventing  the  cancellation 
of  a  program — seem  more  important.  I'hose  imperaiive.s  may  indeed  be  more 
important,  but  it  should  be  clearly  understood  that  the  effect  of  such  '.  ioltitions  is 
likely  to  be  both  liciglilened  risk  and  lowered  e;ipcacit\  to  manage  tiiat  risk.  Tlie  n'osi 
important  message  of  this  study  is  that  the  senior  oTiciais  in  the  Air  Force  and  the 
DoD  who  are  charged  with  making  decisions  concerning  strategies  for  developing 
new  systems,  as  well  as  setting  guidance  for  schedule,  cost,  and  performance  targets, 
should  temper  recommendations  brought  forth  by  their  subordinates  with  an 
understanding  of  the  intense  advocacy  environment  that  frequently  shapes  those 
recommendations.  Indeed,  they  should  also  recognize  that  sometimes  their  own 
behavior  shapes  that  of  their  subordinates. 

To  this  broad  conclusion,  we  would  add  two  more  limited  suggestions.  First,  two  of 
the  programs  we  examined,  the  AFE  and  the  MSIP  for  the  F- 16,  seem  to  us  to  provide 
useful  guidelines  for  R&D  management  at  the  program  office  level  for  a  significant 
proportion  of  the  Air  Force’s  acquisition  program.  Second,  several  of  the  programs 
in  our  study  could  be  said  to  have  gotten  "out  of  control,"  in  the  sense  that  they  were 
subjected  to  a  continuing  series  of  reviews,  site  visits,  program  revisions,  congres¬ 
sional  interventions,  and  personnel  changes  that,  in  our  judgment,  often  seemed  to 
have  deleterious  effects  on  the  program  outcome.  VVe  think  that  many  of  the  actions 
that  complex  and  political  bureaucracies  take  when  a  prngm.n  is  in  deep  trouble  arc 
counterproductive,  and  that  the  Air  Force  and  DoD  should  consider  some  form  of 
special  management  for  programs  that  are  experiencing  great  difficulty — something 
akin  to  putting  a  business  into  receivership. 


PROMOTING  LONG-TERM  RELATIONSHIPS  BETVVTEEN  THE  SPO  AND 
CONTRACTOR 

Two  of  the  programs  we  studied  involved  long  relationships  between  an  SPO  and  one 
or  more  contractors.  In  the  case  of  the  AFE,  the  development  began  largely  at  the 
initiative  of  the  Propulsion  SPO,  which  had  continuing  relationships  with  the  two 
major  U.S.  manufacturcis  of  large  aircraft  engines.  Those  relation.ships  were  devel¬ 
oped  in  the  course  of  managing  a  number  of  diverse  engine  developments  as  well  as 
providing  much  of  the  govemmem  oversight  of  the  firms'  independent  rcsca,  h  and 
development  (IR&D)  activities.  This  cx|  ience  provided  ,Mr  Forec  personnel  with 
ample  opportunity  to  understand  the  opportunities  presented  by  the  technical  ca- 
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pabilities  of  the  two  contractors  and  their  relative  market  positions.  The  develop¬ 
ment  of  the  AVE  seems  to  represent  an  effective  example  of  the  benefits  of  what  we 
term  matiaged  competition. 

The  F-IC  MSir  program  presents  a  different  but  related  picture.  Here,  the  continuity 
of  an  SPO  that  has  been  in  existence  for  more  than  20  years,  coupled  with  an  asso¬ 
ciation  with  a  single  prime  contractor,  permitted  the  development  of  a  well-consid¬ 
ered  process  for  the  incorporation  of  important  and  varied  modifications  in  the  F-16 
system. 

The  program  offices  share  some  important  qualities.  Their  longevity— amounting 
almost  to  permanence — permitted  the  offices  to  develop  substantial  technical  capa¬ 
bility.  Much  of  this  capability  lies  in  a  cadre  of  career  civil  servants,  but  eftectiv'e  and 
well-qualified  uniformed  officers  appear  to  have  been  regularly  assigned  to  and  to 
have  effectively  led  these  offices.  The  size  of  the  programs  managed  by  these  SPOs, 
coupled  with  theit  diverse  responsibilities,  permitted  the  development  of  a  form  of 
matrix  organization  that  permits  the  retention  of  staff  with  substantial  specialized 
expertise  who  work  on  several  programs.  This  appears  to  promote  transfer  of  learn¬ 
ing  from  one  project  to  another.'*  Doctrine  has  been  developed  based  on  experience; 
tlie  Propulsion  ,SI’0  in  jiarticular  had  developed  a  dcvolopncnt  doctrine  tliat  guit'ed 
die  activities  of  its  own  staff  and  tliat  of  die  contractors.  In  me  case  of  the  .TTL,  die 
engines  used  derivative  designs  building  on  earlier  engine  designs,  an  important 
component  of  both  the  SPO  and  contractor  development  doctrines.  The  new 
versions  of  the  F-16  can  similarly  be  seen  as  derivative  designs. 

These  important  internal  qualities  of  the  SPO  are  reinforced  by  the  stable,  long-term 
relationships  it  possesses  with  its  contractors.  Program  managers  can  accumulate  a 
substantial  understanding  of  the  contractors  they  deal  with.  This  permits  far  more 
informed  and  effeedve  responses  to  the  inevitable  surprises  that  occur  in  the  course 
of  a  development.  These  relationships  are  supported  by  the  use  of  "relational  con¬ 
tracts”  that  emphasize  whole  rather  than  component  tasks  and  cat,  e  put  in  place 
and  modified  quickly  as  the  situation  dictates.  The  m.inaged  competition  I  uvecn 
General  Electric  and  Pratt  and  V'liitney  provides  a  good  example  of  how  such  com¬ 
petition  ■  .1'  bo  used  in  conjunction  with  relational  contracts  to  achieve  suits  that 
seens  '-.c.nis  to  have  been  beneficial  to  the  Air  Force  ar.d  the  nation.  Howsvci.  the 
impO'  ti  ;.  of  experienced  and  qualified  SPO  staff  cannot  be  too  strongly  emptia- 
sized.  V'  f,’  CIO  not  believe  tl.ese  policies  will  work  in  a  turbulent  and  unstable  organi¬ 
zational  environment. 

The  Air  Force  has  made  several  decisions  that  lessen  the  opportunity  to  use  this  style 
of  management  for  engines  over  the  past  years.  For  security  reasons,  the  B-2  engines 
were  removed  from  the  purxiew  of  the  Propulsion  SPO,  ano  the  engine  for  the 
advanced  lacti'tal  fighter  (ATF,  now  the  F-22)  was  made  the  responsibility  of  the  .ATF 
SPO,  apparently  for  reasons  of  economy  and  management  control.  We  have  not 
examined  tiiese  decisions  and  cannot  judge  their  appropriateness.  But,  given  the 
evolving  defense  acquisition  environment,  we  think  the  benefits  of  the  rype  of 
acquisition  practices  used  by  the  Pn  pulsion  SPO  (and  the  F-16  SPO)  deserve  serious 
consideration  for  use  in  other  areas. 


'  this  matrl):  actually  exti-nds  lo  elements  of  the  larj^er  technical  staff  in  the  .■Veronaitticai  Systetns  Diviiiot! 
(now  the  Aeronau'icai  Systems  Ontcr ;  as  well. 
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Obviously,  with  the  perceived  reduction  of  military  threats  attendant  to  the  new 
world  c“der,  defense  budgets  have  fallen  and  will  continue  to  fall,  .Moreover,  the  na¬ 
ture  of  the  threat  I'self  has  become  less  certain  and  more  contingencies  need  to  be 
hedged  against,  The  lowered  defense  spending  is  likely  to  lead  to  fewer  competitors 
and  to  a  eonce.m  that  the  nation  will  have  too  few  firms  to  maintain  the  type  of  com¬ 
petition  that  experience  suggests  is  necessary  to  spur  innovation  and  promote  a  de¬ 
gree  of  efficiency.  In  this  situation,  sustained  relationships,  emphasizing  derivative 
designs,  and  the  use  of  managed  competition  and  relational  contracts  become  more 
attractive.  Acquisition  policies  that  use  competitions  that  are  life-and-death  affairs 
for  contractors  (or  dhisions  of  contractors)  are  not  likely  to  be  either  efficient  or  ef¬ 
fective  in  the  new  acquisition  ensdronment.  Important  system  elements,  such  as 
ordnance,  munitions,  and  various  elements  of  avionics,  seem  potential  candidates 
for  such  management. 

SPECIAL  MANAGEMENT  FOR  OUT  OF  CONTROL  PROGRAMS 

Existing  procedures  and  bureaucratic  incentives  do  not  deal  well  with  programs  that 
have  gotten  into  serious  difficulty,  as  was  the  case  with  LW'TIRN'  and  .AMR.AA.M. 
They  lead  to  the  imposition  oi  tighti'r  regulation  and  control  when  flexibility  is  what 
is  needed.  They  suppress  the  discussion  of  iincertairity  when  openness  is  needed. 
Thoy  often  focus  on  political  concern.s  when  the  emphasis  should  be  on  technical 
.,nd  management  issues. 

The  Air  Force  and  DoD  may  want  to  consider  using  a  special  management  unit  and 
reporting  structure  for  a  program  when  it  is  judged  to  have  gone  out  of  control. 
Somewhat  like  a  receivership  for  a  business,  the  unit  would  take  a  hard  and  inde¬ 
pendent  look  at  the  problem,  waive  existing  rules  and  requirements  where  appro¬ 
priate,  and  make  changes  in  both  the  design  and  management  strategy.  Special  re¬ 
porting  channels  and  procedures  would  be  required,  but  great  authority  would  be 
delegated  to  the  indKidual  charged  txaiii  managing  the  program.  He  or  she  would  be 
provided  with  clear  and  explicit  guidance  concerning  the  essential  program  parame¬ 
ters  within  which  he  or  she  must  woik  and  authority  to  make  needed  tradeoffs  widiin 
those  paiameters. 

Implementing  a  policy  such  as  this  would  be  difficult.  Criteria  for  identifying  such 
programs  are  needed,  "’’he  regular  command  structure  can  normally  be  expected  to 
resist  such  designation  because  it  will  be  widtlv  seen  as  an  admission  of  failure  in 
performance.  The  prcc  dures  must  be  worked  out  with  the  Congress  as  they  have 
been  the  source  of  many  of  the  rigidities  that  inhibit  program  recovery.  Thei.‘  legiti¬ 
mate  needs  and  concerns  must  be  addres.sc d  by  the  actions,  and  they  should  concur 
in  them. 

The  most  effective  w'ay  to  deal  with  out  of  control  programs,  of  course,  is  to  prevent 
then;  from  occurring.  The  organizational  barriers  to  effective  prevention  have  been 
the  principal  subject  of  this  study.  Understanding  and  respect  for  these  systemic 
barriers  to  effective  development  ot  systems  on  the  pan  of  all  participants  is  the  best 
hope  for  improved  effectiveness  in  R&C  management  of  large  weapon  systems. 


